Quick Task: How to count the documents of a user via the Shell?

Published on 18 March 2019 in - 2 minutes read - Last modified on 06 March 2021

How to count the documents of a user?

Sometimes, it is convenient to administrate Hue directly via the command line. While investigating while http://demo.gethue.com was slow, we discovered that the demo user had more than 85 000 documents! This was a quick way to validate this and delete the extra ones.

 

On the command line:

.<span class="hljs-meta-keyword">/build/</span>env<span class="hljs-meta-keyword">/bin/</span>hue shell

If using Cloudera Manager, as a root user launch the shell.

Export the configuration directory:

<span class="hljs-built_in">export</span> HUE_CONF_DIR=<span class="hljs-string">"/var/run/cloudera-scm-agent/process/`ls -alrt /var/run/cloudera-scm-agent/process | grep HUE_SERVER | tail -1 | awk '{print <span class="hljs-variable">$9</span>}'`"</span>
<span class="hljs-built_in">echo</span> <span class="hljs-variable">$HUE_CONF_DIR</span>
> /var/run/cloudera-scm-agent/process/2061-hue-HUE_SERVER

Get the process id:

<span class="hljs-attribute">lsof</span> -i :<span class="hljs-number">8888</span>|grep -m1 hue|awk <span class="hljs-string">'{ print <span class="hljs-variable">$2</span> }'</span>
> <span class="hljs-number">14850</span>

In order to export all Hue’s env variables:

<span class="hljs-keyword">for</span> line <span class="hljs-keyword">in</span> `strings /proc/$(lsof -i :8888|grep -m1 hue|awk <span class="hljs-string">'{ print $2 }'</span>)/environ|egrep -v <span class="hljs-string">"^HOME=|^TERM=|^PWD="</span>`;<span class="hljs-keyword">do</span> <span class="hljs-built_in">export</span> <span class="hljs-variable">$line</span>;<span class="hljs-keyword">done</span>

And finally launch the shell by:

HUE_IGNORE_PASSWORD_SCRIPT_ERRORS=<span class="hljs-number">1</span> <span class="hljs-regexp">/opt/</span>cloudera<span class="hljs-regexp">/parcels/</span>CDH<span class="hljs-regexp">/lib/</span>hue<span class="hljs-regexp">/build/</span>env<span class="hljs-regexp">/bin/</span>hue shell
> <span class="hljs-string">ALERT:</span> This appears to be a CM Managed environment
> <span class="hljs-string">ALERT:</span> HUE_CONF_DIR must be set when running hue commands <span class="hljs-keyword">in</span> CM Managed environment
> <span class="hljs-string">ALERT:</span> Please run <span class="hljs-string">'hue <command> --cm-managed'</span>

Then use the Python code to access a certain user information:

Python <span class="hljs-number">2.7</span>.<span class="hljs-number">6</span> (default, Oct <span class="hljs-number">26</span> <span class="hljs-number">2016</span>, <span class="hljs-number">20</span>:<span class="hljs-number">30</span>:<span class="hljs-number">19</span>)
Type <span class="hljs-string">"copyright"</span>, <span class="hljs-string">"credits"</span> or <span class="hljs-string">"license"</span> <span class="hljs-keyword">for</span> more information.

IPython <span class="hljs-number">5.2</span>.<span class="hljs-number">0</span> -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython<span class="hljs-string">'s features.
%quickref -> Quick reference.
help      -> Python'</span>s own help system.
object?   -> Details about <span class="hljs-string">'object'</span>, use <span class="hljs-string">'object??'</span> <span class="hljs-keyword">for</span> extra details.

from django.contrib.auth.models import User
from desktop.models import Document2

user = User.objects.get(username=<span class="hljs-string">'demo'</span>)
Document2.objects.documents(user=user).count()

<span class="hljs-keyword">In</span> [<span class="hljs-number">8</span>]: Document2.objects.documents(user=user).count()
Out[<span class="hljs-number">8</span>]: <span class="hljs-number">1167</span>

<span class="hljs-keyword">In</span> [<span class="hljs-number">10</span>]: Document2.objects.documents(user=user, perms=<span class="hljs-string">'own'</span>).count()
Out[<span class="hljs-number">10</span>]: <span class="hljs-number">1166</span>

<span class="hljs-keyword">In</span> [<span class="hljs-number">11</span>]: Document2.objects.documents(user=user, perms=<span class="hljs-string">'own'</span>, include_history=True).count()
Out[<span class="hljs-number">11</span>]: <span class="hljs-number">7125</span>

<span class="hljs-keyword">In</span> [<span class="hljs-number">12</span>]: Document2.objects.documents(user=user, perms=<span class="hljs-string">'own'</span>, include_history=True, include_trashed=True).count()
Out[<span class="hljs-number">12</span>]: <span class="hljs-number">7638</span>

<span class="hljs-keyword">In</span> [<span class="hljs-number">13</span>]: Document2.objects.documents(user=user, perms=<span class="hljs-string">'own'</span>, include_history=True, include_trashed=True, include_managed=True).count()
Out[<span class="hljs-number">13</span>]: <span class="hljs-number">31408</span>

Out[<span class="hljs-number">14</span>]:
(<span class="hljs-number">85667</span>L,
{u<span class="hljs-string">'desktop.Document'</span>: <span class="hljs-number">18524</span>L,
  u<span class="hljs-string">'desktop.Document2'</span>: <span class="hljs-number">31409</span>L,
  u<span class="hljs-string">'desktop.Document2Permission'</span>: <span class="hljs-number">556</span>L,
  u<span class="hljs-string">'desktop.Document2Permission_groups'</span>: <span class="hljs-number">277</span>L,
  u<span class="hljs-string">'desktop.Document2Permission_users'</span>: <span class="hljs-number">0</span>L,
  u<span class="hljs-string">'desktop.Document2_dependencies'</span>: <span class="hljs-number">15087</span>L,
  u<span class="hljs-string">'desktop.DocumentPermission'</span>: <span class="hljs-number">1290</span>L,
  u<span class="hljs-string">'desktop.DocumentPermission_groups'</span>: <span class="hljs-number">0</span>L,
  u<span class="hljs-string">'desktop.DocumentPermission_users'</span>: <span class="hljs-number">0</span>L,
  u<span class="hljs-string">'desktop.Document_tags'</span>: <span class="hljs-number">18524</span>L})

Et voila, it is much cleaner now!


comments powered by Disqus

More recent stories

15 November 2021
Create SQL tables from excel files
Read More
21 September 2021
Access your data in ABFS without any credential keys!
Read More