Quick Task: How to count the documents of a user via the Shell?

18 March 2019 in Administration - 2 minutes read

How to count the documents of a user?

Sometimes, it is convenient to administrate Hue directly via the command line. While investigating while http://demo.gethue.com was slow, we discovered that the demo user had more than 85 000 documents! This was a quick way to validate this and delete the extra ones.

 

On the command line:

.<span class="hljs-meta-keyword">/build/</span>env<span class="hljs-meta-keyword">/bin/</span>hue shell

If using Cloudera Manager, as a root user launch the shell.

Export the configuration directory:

<span class="hljs-built_in">export</span> HUE_CONF_DIR=<span class="hljs-string">"/var/run/cloudera-scm-agent/process/`ls -alrt /var/run/cloudera-scm-agent/process | grep HUE_SERVER | tail -1 | awk '{print <span class="hljs-variable">$9</span>}'`"</span>
<span class="hljs-built_in">echo</span> <span class="hljs-variable">$HUE_CONF_DIR</span>
> /var/run/cloudera-scm-agent/process/2061-hue-HUE_SERVER

Get the process id:

<span class="hljs-attribute">lsof</span> -i :<span class="hljs-number">8888</span>|grep -m1 hue|awk <span class="hljs-string">'{ print <span class="hljs-variable">$2</span> }'</span>
> <span class="hljs-number">14850</span>

In order to export all Hue’s env variables:

<span class="hljs-keyword">for</span> line <span class="hljs-keyword">in</span> `strings /proc/$(lsof -i :8888|grep -m1 hue|awk <span class="hljs-string">'{ print $2 }'</span>)/environ|egrep -v <span class="hljs-string">"^HOME=|^TERM=|^PWD="</span>`;<span class="hljs-keyword">do</span> <span class="hljs-built_in">export</span> <span class="hljs-variable">$line</span>;<span class="hljs-keyword">done</span>

And finally launch the shell by:

HUE_IGNORE_PASSWORD_SCRIPT_ERRORS=<span class="hljs-number">1</span> <span class="hljs-regexp">/opt/</span>cloudera<span class="hljs-regexp">/parcels/</span>CDH<span class="hljs-regexp">/lib/</span>hue<span class="hljs-regexp">/build/</span>env<span class="hljs-regexp">/bin/</span>hue shell
> <span class="hljs-string">ALERT:</span> This appears to be a CM Managed environment
> <span class="hljs-string">ALERT:</span> HUE_CONF_DIR must be set when running hue commands <span class="hljs-keyword">in</span> CM Managed environment
> <span class="hljs-string">ALERT:</span> Please run <span class="hljs-string">'hue <command> --cm-managed'</span>

Then use the Python code to access a certain user information:

Python <span class="hljs-number">2.7</span>.<span class="hljs-number">6</span> (default, Oct <span class="hljs-number">26</span> <span class="hljs-number">2016</span>, <span class="hljs-number">20</span>:<span class="hljs-number">30</span>:<span class="hljs-number">19</span>) 
Type <span class="hljs-string">"copyright"</span>, <span class="hljs-string">"credits"</span> or <span class="hljs-string">"license"</span> <span class="hljs-keyword">for</span> more information.

IPython <span class="hljs-number">5.2</span>.<span class="hljs-number">0</span> -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython<span class="hljs-string">'s features.
%quickref -> Quick reference.
help      -> Python'</span>s own help system.
object?   -> Details about <span class="hljs-string">'object'</span>, use <span class="hljs-string">'object??'</span> <span class="hljs-keyword">for</span> extra details.

from django.contrib.auth.models import User
from desktop.models import Document2

user = User.objects.get(username=<span class="hljs-string">'demo'</span>)
Document2.objects.documents(user=user).count()

<span class="hljs-keyword">In</span> [<span class="hljs-number">8</span>]: Document2.objects.documents(user=user).count()
Out[<span class="hljs-number">8</span>]: <span class="hljs-number">1167</span>

<span class="hljs-keyword">In</span> [<span class="hljs-number">10</span>]: Document2.objects.documents(user=user, perms=<span class="hljs-string">'own'</span>).count()
Out[<span class="hljs-number">10</span>]: <span class="hljs-number">1166</span>

<span class="hljs-keyword">In</span> [<span class="hljs-number">11</span>]: Document2.objects.documents(user=user, perms=<span class="hljs-string">'own'</span>, include_history=True).count()
Out[<span class="hljs-number">11</span>]: <span class="hljs-number">7125</span>

<span class="hljs-keyword">In</span> [<span class="hljs-number">12</span>]: Document2.objects.documents(user=user, perms=<span class="hljs-string">'own'</span>, include_history=True, include_trashed=True).count()
Out[<span class="hljs-number">12</span>]: <span class="hljs-number">7638</span>

<span class="hljs-keyword">In</span> [<span class="hljs-number">13</span>]: Document2.objects.documents(user=user, perms=<span class="hljs-string">'own'</span>, include_history=True, include_trashed=True, include_managed=True).count()
Out[<span class="hljs-number">13</span>]: <span class="hljs-number">31408</span>

Out[<span class="hljs-number">14</span>]:
(<span class="hljs-number">85667</span>L,
{u<span class="hljs-string">'desktop.Document'</span>: <span class="hljs-number">18524</span>L,
  u<span class="hljs-string">'desktop.Document2'</span>: <span class="hljs-number">31409</span>L,
  u<span class="hljs-string">'desktop.Document2Permission'</span>: <span class="hljs-number">556</span>L,
  u<span class="hljs-string">'desktop.Document2Permission_groups'</span>: <span class="hljs-number">277</span>L,
  u<span class="hljs-string">'desktop.Document2Permission_users'</span>: <span class="hljs-number">0</span>L,
  u<span class="hljs-string">'desktop.Document2_dependencies'</span>: <span class="hljs-number">15087</span>L,
  u<span class="hljs-string">'desktop.DocumentPermission'</span>: <span class="hljs-number">1290</span>L,
  u<span class="hljs-string">'desktop.DocumentPermission_groups'</span>: <span class="hljs-number">0</span>L,
  u<span class="hljs-string">'desktop.DocumentPermission_users'</span>: <span class="hljs-number">0</span>L,
  u<span class="hljs-string">'desktop.Document_tags'</span>: <span class="hljs-number">18524</span>L})

Et voila, it is much cleaner now!


comments powered by Disqus

More recent stories

13 November 2019
Visually surfacing SQL information like Primary Keys, Foreign Keys, Views and Complex Types
Read More
31 October 2019
Missing some color? How to improve or add your own SQL syntax Highlighter
Read More
24 October 2019
How to create a HBase table on Kerberized Hadoop clusters
Read More