Matias Carrasco Kind @ NCSA

DES Collaboration Phone Call, September 12th, 2016

Data Access for Astronomical Surveys

A brief description and demo of tools @ DESDM

DES Collaboration Phone Call

Matias Carrasco Kind @ NCSA/UIUC


  • DES : Gold (Data) Mine

  • Excellent Job at mining the data

  • Consumer outside the mine

  • Need to bring gold (data) outside

  • Tools and interfaces

  • Getting ready for Y3!


DES Science Portal

  • @ NCSA by managed by LIneA

  • Interactive Coadd Image exploration

  • Moving some services from old-portal to the Science portal

  • Already uploaded with Y3A1 Test data




  • DES DB in Oracle

  • Needed upgrade from older versions (trivialaccess)

  • Enhanced SQL command line interpreter

  • Astronomically friendly

  • Python API, web interface

  • There are many other clients but needed one standard for beginners

Some cool features

  • History of commands
  • Smart TAB auto-completion of commands, columns, users, tables and path
  • Very flexible configuration
  • Completely open source and written in python
  • Write tables in csv, tab, fits and hdf5 formats
  • Load sql queries directly from a file
  • Load table directly in to DB in one step, no memory issues
  • Can be imported as python module (py2/py3)
  • Tables are handled and displayed using pandas
  • Can run python functions inside query (beta)
  • Many more...


Making things even easier

  • Clone it from github
  • Using pip or easy_install
  • Using EUPS @ DESDM
  • Pre-installed in Docker Image (and run outside the container)
  • New! Install with conda

conda install easyaccess -c mgckind

Future and Demo

  • Generalization (SQLAlchemy), in-query python functions, batch jobs, checkpoint, web, better Jupyter integration, more...

  • easyacess: Enhanced SQL interpreter for astronomical surveys (Carrasco Kind+, 2016?)

  • For a more detailed tutorial check Here


Some DESDM statistics

(Scroll down )

DESDM New accounts

More than 220 New users since new On-boarding!

New DES Member? Check Here

Several Services: DB, Confluence, HipChat, BitBucket, Stackoverflow

DB Users

Doubled number of users since easyaccess

More than 180 users with easyaccess with different activity levels

Feedback via github issues, stack-overflow-like questions, email list and personal inquiries

DB Quotas

Personal space actively used and constantly increasing

People create and share tables with one command

~ 4.5 TB in personal space, tables and index tables included

SV Release

Experimental UI projects being developed (Openstack + Docker)

Experimental UI projects being developed (Openstack + Docker)

JupyterHub + easyaccess + DS9

  • Multi-user jupyter notebook sessions
  • Access with DB credentials
  • Running in OpenStack ("The Cloud")
  • Easy setup (Running from Docker container)
  • Easyaccess and other tools installed
  • Local JS9 (DS9 in Javascript) integration
  • Still in beta mode, planning to scale up
  • Open to DES collaborators
Live demo!

Online Cutout Services

  • Cutout Services for both: Coadd images and Single Epoch

  • Nice REST API so can be also accessed from e.g. python

  • Almost production-like services with GPFS mounted

  • Soon to be "oficcially" released, we are merging both services

Live demo!

Coadd image cutouts example

Integrated tools demo


  • Tools being developed to make data access easier

  • Easyaccess, Cutouts, Jupyter, Js9, Science Portal, etc...

  • Opensource and integrated software

  • Bring software to the data!

  • Want to help? Want some infrastructure? Let me know!



Matias Carrasco Kind