Thursday, November 26, 2015

Calling 3rd Party Python Modules in Blender Scripts

Introduction

This costed me nerves....
Yesterday I had the idea of connecting a Blender 2.76b via a Blender plugin - which still needs to be written... - to a SQL Server database installation where I keep geometric data of some objects. Furthermore I am working in a windows environment using a Anaconda python installation, which I normally run with Python 2.7.6.
The python weapon of choice to connect to a database is the module PYODBC which offers convenient functions to access databases. Unfortunately, this module is not available in Blender's bundled Python installation. I did not manage to install pyodbc within the context of Blender's bundled python

So this is where my odysee started. I had the following challenges to solve:
  • Create a Python environment compatible woth Blender 2.76b
  • Turn Blender's python environment to the environment with pydbc

 Creating an Independent Python Environment

As I said I am using the Anaconda IDE. My first approach was to install plain Python 3.4.2 (required for Blender 2.76) from python.org in parallel to Anaconda. Installation and stuff wen fine, but quickly I ran into the issue that I wanted to install the pyodbc package on top of the standard installation and this was not possible as this installation requires a working VS2013 (?) environment to compile the sources. This is where I went back to Anaconda.
In Anaconda it is possible to create version dependent python environments in parallel to the intitial installation using the command
  • conda create -n <environment name> python=<version number>
The new environment can be found at the path:
  • ..\<Anaconda root path>\envs\<environment name>

To activate the environment call in the command line:
  • activate <environment name>
So whatever you have to change to adjust your Python installation for Blender needs to be done in the environment with the right version number for Blender. The following steps assume that you activated an environment with this right version.
In Anaconda modules like pyodbc can be installed very easily without the need for external tools and compilers. This is done by calling (in this example I am installing pyodbc):
  • conda install pyodbc
The conda command  fetches all the required sources from the internet, compiles them if needed and adjusts the environment so they can be used.
3rd party modules (like pyodbc) are stored at a path
  • ..\<Anaconda root path>\envs\<environment name>\Lib\site-packages
This is where Blender needs to look for the 3rd party module.

Telling Blender where to Find Things

I tried a couple of things to tell Blender where to look for pyodbc, including setting the PYTHONPATH variable following this blog post, deleting Blender's Python, etc. However, all of these approaches were very cumbersome and led to negative interferences between Blender and Anaconda - meaning either of these did not work.
Finally I found a solution which seems to me quite elegant: I don't do any system-related settings like PYTHONPATH and stuff and do everything need in my script.
I simple check if the path to pyodbc (or any other 3rd party module) is already included in the sys.path varialble of Python. If this is not the case I append the required path and import the module after having registered the path:

import sys

pyodbcPath = '..\\<Anaconda root path>\\envs\\<environment name>\\Lib\\site-packages'
systemPaths = sys.path

checkResult = [s for s in systemPaths if pyodbcPath in s]

#check if path to pyodbc exists in sys and add if needed
if (checkResult == []):
    sys.path.append(pyodbcPath)
   
import pyodbc
import bpy

[...] Your code comes here

This is it.

Thanks for reading my post, I hope you enjoyed it and it helps you with your own project.
Keep on geekin'! Your
WolfiG


Wednesday, November 25, 2015

Building Data Driven Applications: Combining Vis.js with Paper.js

Introduction

Currently I am working on a visualization project where I want display real world entities with the help of a network graph. Additional to the classical network graph and graphics I want to display information like groups of network nodes, or I want to highlight groups of nodes based on node attributes. Here an example of combined information:
The network graph displays the connection of "things"/entities and the colored areas show groups of these entities.
All the data I am using is stored in a database, so I was looking for a way to visualize these data with the least effort and media breaks. Furthermore, one factor for the choice of tools was the possibility to deploy visualized data with the least possible effort and in best case as view on "real time" data in the database.

Frameworks for Network Graphs

To achieve my task I evaluated a couple of tools and frameworks. My first choice was GEPHI, which is available in version 0.8.2. GEPHI provides means to layout quite large numbers of nodes and edges into nicely looking network graphs and it is quite popular among scientists and practitioners using networks. However, I found a couple of drawbacks which made me to leave the. GEPHI path: first of all GEPHI is a closed, standalone program. Hence, it is hard to enhance a readily layouted graph with additional information and this requires additional external tools. Furthermore it is not possible to provide interactive, real time graphs which can be presented to an end user. The GEPHI workflow requires the following steps: draw data from database (GEPHI has a direct DB interface), layouting of the network with some algorythm, export visualization to a format which allows post processing - e.g. SVG - and do the actual post processing.
Finaly I decided to abandon the GEPHI path as it involved too many steps with human intervention and turned to web-based tools. Quite quickly I found javascript frameworks as my weapon of choice and evaluated the following frameworks for network layouting: D3.js, cytoscape.js,  sigma.js and vis.js. To keep things short, I ended up with Vis.js, mostly because it was the only framework which allowed me to pin down nodes via configuration file (a JSON I export from database) and because it provides a "force atlas 2"-like layouting algorithm which yields networks layouts I was aiming for.

Frameworks for HTML5 Graphics

For the "additional" graphics frameworks I ended up quickly with the "big three": Paper.js, Processing.js and Raphael.js. I started with Raphael.js which I knew from a project I was involved before, but it quickly turned out that Raphael's SVG cannot or is hard to combine with vis.js as the latter one uses the HTML5 canvas. I didn't get warm with Processing.js as it uses its own scripting language, so finally I ended up with Paper.js, which for my taste yields satisfactory results.

Vis.js' Door to other Frameworks

The first steps of combining vis.js and paper.js were not easy. I managed to overcome the first obstacles with the help of the vis.js team (thanks for that). Based on one of the vis.js network examples I was able to hook Paper.js into Vis.js.

As the first step you have to tell Paper.js to paint on the Vis.js canvas. This can be done in the main program by the following lines:
[...] // Things required to setup network, etc.
// "mynetwork" is defined in html <body>
var container = document.getElementById('mynetwork');
var network = new vis.Network(container, data, options);
[...] // More network stuff here if needed
paperCanvas = container.firstChild.canvas;
paper.setup(paperCanvas);

The main entry points to inject other graphics into the Vis .js canvas are the Vis.js network events
  • network.on("beforeDrawing", function(ctx) { [...] }) this will draw graphics behind the network graph
  • network.on("afterDrawing", function(ctx) { [...] }) this will draw in front of th network graph
With these hook events one can start to draw Paper.js elements and display them together with the network graph. You can put any Paper.js related code replacing the square brackets. I use Vis.js Network/DataSet methods to extract geometric data on positions, etc. and pass them to Paper.js to draw elements related to the network nodes, etc.

I hope this posts helps you with your own project. Keep on geekin', your
WolfiG