Kivy Garden - Fixed to fertile land again

Kivy fascinates me when it comes to a python UI framework. However while I was installing the pre-requisites I was looking for a kivy-designer which is thankfully available on github.

https://github.com/kivy/kivy-designer

This chap requires the garden and filebrowser class from the kivy garden which is suppose to be downloaded by the garden script which is nothing but a python script. The moment I started setting up the bloke I get

$ garden install filebrowser
Downloading http://github.com/kivy-garden/garden.filebrowser/archive/master.zip ...
Traceback (most recent call last):
  File "/home/addy/workspace/kivyenv/bin/garden", line 190, in <module>
    GardenTool().main(sys.argv[1:])
  File "/home/addy/workspace/kivyenv/bin/garden", line 72, in main
    options.func()
  File "/home/addy/workspace/kivyenv/bin/garden", line 110, in cmd_install
    fd = self.download(opts.package)
  File "/home/addy/workspace/kivyenv/bin/garden", line 171, in download
    data += buf
TypeError: Can't convert 'bytes' object to str implicitly

The problem is with the download call. The initial data buffer is initialized as a str object. The 'r' is a variable which contains the request library's download request data which would a sequence of bytes.  Ultimately the goal is to pass the in-memory binary streams as a zipfile. However python 3 will not let us convert the bytes to str directly.  Observe the following code 

    def download(self, package):

        url = 'http://github.com/kivy-garden/{}/archive/master.zip'.format(

                package)

        print('Downloading {} ...'.format(url))

        r = requests.get(url)#, prefetch=False) # r is requests library's download content 

        if r.status_code != 200:

            print('Unable to found the garden package. (error={})'.format(

                r.status_code))

            sys.exit(1)

        animation = '\\|/-'

        index = 0

        count = 0

        data = '' # problem 1 

#the loop is basically re-assembling the binary stream 

        for buf in r.iter_content(1024):

            index += 1

            data += buf  ## this is what is throwing the exception 

            count += len(buf)

            print('Progression', count, animation[index % len(animation)], '\r')

            sys.stdout.flush()

        print('Download done ({} downloaded)'.format(count))

        

        return StringIO(data) ## why aren't we returning binary data 

Potential fix to the problem which finally worked 

# add the import to the top of the file. 

    from io import BytesIO

    def download(self, package):

        url = 'http://github.com/kivy-garden/{}/archive/master.zip'.format(

                package)

        print('Downloading {} ...'.format(url))

        r = requests.get(url)#, prefetch=False)

        if r.status_code != 200:

            print('Unable to found the garden package. (error={})'.format(

                r.status_code))

            sys.exit(1)

        animation = '\\|/-'

        index = 0

        count = 0

        data = b'' #this takes care of the string conversion

        for buf in r.iter_content(1024):

            index += 1

            data += buf  # no more error here 

            count += len(buf)

            print('Progression', count, animation[index % len(animation)], '\r')

            sys.stdout.flush()

        print('Download done ({} downloaded)'.format(count))

        

        return BytesIO(data) #return a byte stream than a string stream 

Wallah !! all works now !! 

$ garden install filebrowser

Downloading http://github.com/kivy-garden/garden.filebrowser/archive/master.zip ...

Progression 1024 | 

...... snppped ... 

Progression 206959 - 

Download done (206959 downloaded)

Extracting...

Installing new version...

Done! garden.filebrowser is installed at: /home/addy/.kivy/garden/garden.filebrowser

Cleaning...

What Happens when we don't read the manual

This is RTFD kind of a pun which happened with me. Shooting self in the foot is more like it. I was trying to play around with ctypes and was trying to figure out how to get the last error code. I didn't read the python docs man page till this point and kept doing

#----------------python3 so don't be amazed when we do conversions-----------------------
import ctypes
import ctypes.util
libc = ctypes.util.find_library("c")
libc.prinf("Hello world") # this is not going to print nothing
errno = libc.errno
print( "Error code is " + str(errno))

#------------------------------

Guess what I get , a SIGSEGV .. !!!!! WOW and I now become more bold and try to get deeper with gdb. So here is how I prove myself to be a member of all idiotic morons society's prime member

 gdb python3
GNU gdb (Ubuntu 7.7.1-0ubuntu5~14.04.2) 7.7.1

Reading symbols from python3...Reading symbols from /usr/lib/debug//usr/bin/python3.4...done. #download the python symbols for this
done.
(gdb) run
Starting program: /usr/bin/python3
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Python 3.4.3 (default, Oct 14 2015, 20:28:29)
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes
>>> import ctypes.util
>>> libc = ctypes.util.find_library("c")
>>> libc.printf("Hello World")
>>> libc = ctypes.CDLL(libc)
>>> libc.printf("Hello World")
1
H>>> libc.errno
<_FuncPtr object at 0x7ffff5e58688>
>>> libc.errno()

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff7fd16a0 in ?? ()
(gdb) bt 5
#0  0x00007ffff7fd16a0 in ?? () <== null function
#1  0x00007ffff5f49adc in ffi_call_unix64 () from /usr/lib/x86_64-linux-gnu/libffi.so.6
#2  0x00007ffff5f4940c in ffi_call () from /usr/lib/x86_64-linux-gnu/libffi.so.6
#3  0x00007ffff6159fed in _call_function_pointer (argcount=0, resmem=0x7fffffffdaf0, restype=<optimized out>, atypes=<optimized out>, avalues=0x7fffffffdae0,
    pProc=0x7ffff7fd16a0, flags=4353) at /build/python3.4-eNHHUi/python3.4-3.4.3/Modules/_ctypes/callproc.c:811
#4  _ctypes_callproc (pProc=pProc@entry=0x7ffff7fd16a0, argtuple=argtuple@entry=(), flags=4353, argtypes=argtypes@entry=0x0,
    restype=<_ctypes.PyCSimpleType at remote 0xa99178>, checker=0x0) at /build/python3.4-eNHHUi/python3.4-3.4.3/Modules/_ctypes/callproc.c:1149
(More stack frames follow...)
(gdb)




finally reading through the documentation the appropriate way of reading the error number is :



Python 3.4.3 (default, Oct 14 2015, 20:28:29)
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes
>>> import ctypes.util
>>> libc = ctypes.util.find_library("c")
>>> libc = ctypes.CDLL(libc)
>>> libc.printf("Hello World")
>>> ctypes.get_errno()

mount and umount with ctypes

Automating a linux build I came across a phase where we had to do mount. A simple way was to perform a mount command using the subprocess module as shown in the code snippet below

#------------------------
import subprocess

mountSource =  "/home/addy/workspace/pymount/test1"
mountTarget =   "/home/addy/workspace/pymount/test2"
fstype = "ext4"

command = "mount -B -t {2} {0} {1}".format(mountSource, mountTarget,fstype) 

print(command.split(" "))
output = subprocess.check_output(command.split(" "))

#-------------------------

or we could have used ctypes. Using python3 the challenge was to make ctypes work. Python3 strings are unicode by default. We need to convert them to ASCII for linux system calls via the ctypes library.

Although this was an easy task once we understand the intrinsics. We should use encode method of the strings to convert them to the ascii strings.

Following code demonstrates the mount from one directory to another.

#-------------------------
import ctypes
import ctypes.util
import os

mountSource =  "/home/addy/workspace/pymount/test3".encode(encoding='ascii', errors='replace')
mountTarget =   "/home/addy/workspace/pymount/test4".encode(encoding='ascii', errors='replace')

libcPath = ctypes.util.find_library("c")
libc = ctypes.CDLL(libcPath)

retCode = libc.mount(mountSource,mountTarget,None,4096,None)
if retCode != 0:
    #errNumber = libc.errno()
    #print ("Error number from libc " + errNumber)
    print ("os.ErrNo = " + str(os.error() ))

print("ls the mountTarget " + str(mountTarget))
output = os.system("ls -al " + mountTarget.decode("utf-8","ignore"))

errorCode = libc.umount(mountTarget,None)
print("Error Code for umount" + str(errorCode))
if errorCode != 0:
    print("Unmount unsuccessful " + str(libc.errno))
else:
    print("successful unmount")

print("unmount")


#------------------------------------------

A Full class implementation although not tested yet 

#----------------------------------------

import ctypes
import ctypes.util
import os
import pwd

class Mounter(object): 
    """
    sys/mount.h constants
    """
    MS_RDONLY = 1       
    MS_NOSUID = 2      
    MS_NODEV = 4          
    MS_NOEXEC = 8      
    MS_SYNCHRONOUS = 16       
    MS_REMOUNT = 32       
    MS_MANDLOCK = 64       
    MS_DIRSYNC = 128       
    MS_NOATIME = 1024       
    MS_NODIRATIME = 2048       
    MS_BIND = 4096       
    MS_MOVE = 8192
    MS_REC = 16384
    MS_SILENT = 32768
    MS_POSIXACL = 1 << 16   
    MS_UNBINDABLE = 1 << 17   
    MS_PRIVATE = 1 << 18     
    MS_SLAVE = 1 << 19       
    MS_SHARED = 1 << 20       
    MS_RELATIME = 1 << 21   
    MS_KERNMOUNT = 1 << 22   
    MS_I_VERSION =  1 << 23   
    MS_STRICTATIME = 1 << 24   
    MS_ACTIVE = 1 << 30
    MS_NOUSER = 1 << 31
   
   
   
    def __init__(self,sourcePath,destPath,fileSystem = None,flags=Mounter.MS_RDONLY):
       
        if fileSystem != None:
            fslist = []
            with open('/proc/filesystems') as filesystems:
                for line in filesystems:
                    fs = line.split("\t")[1]
                    fslist.append(fs)
        if not fileSystem in fslist:
            raise ValueError([fileSystem," doesn't exist in filesystem list ",fslist])
               
        if not os.access(sourcePath,os.R_OK) :
            raise ValueError(["Cannot access " , sourcePath])
        if not os.access(destPath,os.W_OK):
            raise ValueError(["Cannot access " , destPath])
       
        #since we have reached here
        self.fileSystem = fileSystem
        self.libc = ctypes.CDLL(self.get_libc())
        self.mountSource = sourcePath
        self.mountTarget = destPath
        self.flags = flags
       
       
    def get_libc(self):
        libcPath = ctypes.util.find_library("c")
        return libcPath
   
    def is_current_user_root(self):
        pwDBEntry = pwd.getpwuid(os.getuid())
        userName = pwDBEntry.pw_name
        if userName == "root":
            return True
        else:
            return False
           
    def mount_path(self):
        mountPathBytes = self.mountSource.encode("ascii","ignore")
        targetPathBytes = self.mountTarget.encode("ascii","ignore")
        fileSystemBytes = self.fileSystem.encode("ascii","ignore")
       
        errorCode = self.libc.mount(mountPathBytes,
                                    targetPathBytes,
                                    fileSystemBytes,
                                    self.flags,None) #extra data is considered null
       
        if errorCode != 0:
            osErrorCode = ctypes.get_errno()
            osErrorString = os.errno.errorcode[osErrorCode]
            raise OSError([osErrorCode, osErrorString])
       
   
    def unmount_path(self,flags=None):
        targetPathBytes = self.mountTarget.encode("ascii","ignore")
        errorCode = self.libc.umount(targetPathBytes,flags)
        if errorCode != 0:
            osErrorCode = ctypes.get_errno()
            osErrorString = os.errno.errorcode[osErrorCode]
            raise OSError([osErrorCode, osErrorString])

#-----------------------------------------------------