Matias Carrasco Kind @ NCSA

NCSA Astronomy Survey Meeting , January 26th, 2017

Data Access Tools and Data Releases

A brief description

Matias Carrasco Kind @ NCSA/UIUC

Survey meeting @ NCSA, January 26th



  • DES Survey: Gold (Data) Mine

  • DESDM: Excellent job at mining the data

  • DES: Consumer outside the mine

  • Need to bring gold (data) outside

  • Tools and interfaces

  • Y3A1 is out!


Data Management and operations for DES and LSST

DES Science Portal

  • @ NCSA but managed by LIneA

  • Interactive Coadd Image exploration

  • Moving some services from old-portal to the Science portal

  • Already with Y3A1 data

NOAO Data Lab

  • Also a VM @ NCSA but managed by NOAO

  • New initiave and release partner for DR1

  • Mutual benefits

  • Will keep data private until release




  • DES DB in Oracle

  • Needed upgrade from older versions (trivialaccess)

  • Enhanced SQL command line interpreter

  • Astronomer friendly

  • Python API, web interface

  • There are many other CLI and GUI clients. Needed a simple tool

Some cool features

  • History of commands
  • Smart TAB auto-completion of commands, columns, users, tables and path
  • Very flexible configuration
  • Completely open source and written in python
  • Write tables in csv, tab, fits and hdf5 formats
  • Load sql queries directly from a file
  • Load table directly in to DB in one step, no memory issues
  • Can be imported as python module (py2/py3)
  • Tables are handled and displayed using pandas
  • Can run python functions inside query (beta)
  • Many more...


Making things even easier

  • Clone it from github
  • Using pip or easy_install
  • Using EUPS @ DESDM
  • Pre-installed in Docker Image (and run outside the container)
  • New! Install with conda

conda install easyaccess -c mgckind

Experimental UI projects being developed (Openstack + Kubernetes + Docker)

Experimental UI projects being developed (Openstack + Docker)

JupyterHub + easyaccess + JS9

  • Multi-user jupyter notebook sessions
  • Access with DB credentials
  • Running in OpenStack ("The Cloud")
  • Easy setup (Running from Docker container)
  • Easyaccess and other tools installed
  • Spark Cluster in alpha stage
  • Local JS9 (Next gen. DS9) integration
  • Still in beta mode, planning to scale up
  • Open to DES collaborators

Github Org

Ask me for invite!

Online Cutout Services

  • Cutout Services for both: Coadd images and Single Epoch

  • Nice REST API so can be also accessed from e.g. python

  • Almost production-like services with GPFS mounted

Live demo!

Data Release 1

  • A release is not only data! but interfaces, software, archive, documentation, support and commitment

  • Each component is not a single unit, e.g. Documentation is not only a paper relase but also tutorials, data model, DB, software, etc...

  • Content, validation and vetting done with collaboration

  • Many factors/people involved, long term plan

  • Internal and public release, DES DR1 December 15th, 2017

Public DR1 Plan @ DES-docdb 9410


  • Tools being developed to make data access easier @ LIneA, NOAO and DESDM

  • Easyaccess, Cutouts, Jupyter, Js9, Science Portal, etc...

  • Opensource and integrated software

  • Bring software to the data!

  • Want to help? Want some infrastructure? Let me know!



Matias Carrasco Kind