Producing DL h5 files from aanet h5 files

Describes how to use OrcaSong in python to produce h5 files for Deep Learning from aanet h5 files. These files can contain either images (for convolutional networks), or graphs (for Graph networks).

Mode 1: Producing images
Mode 2: Producing Graphs
- Basic Use
General usage
- Calibration
- Adding mc_info

Mode 1: Producing images 

Generate multidimensional images out of km3net data.

Basic Use 

Import the main class, the FileBinner (see orcasong.core.FileBinner), like this:

from orcasong.core import FileBinner

The FileBinner allows to make nd histograms (“images”) from h5-converted root files. To do this, you can pass a list defining the binning. E.g., the following would set up the file binner to generate zt data:

bin_edges_list = [
    ["pos_z", np.linspace(0, 200, 11)],
    ["time", np.linspace(-50, 550, 101)],
]

fb = FileBinner(bin_edges_list)

Note

You have to calibrate the file if it is not calibrated already (i.e. if you there are no columns like pos_z in the hits).

Calling the object like this will show you the binning:

>>> fb
<FileBinner: ('pos_z', 'time') (10, 100)>

As you can see, the FileBinner will produce zt data, with 10 and 100 bins, respectively. Convert a file like this:

fb.run(infile, outfile)

Or convert multiple files, which will all be saved in the given folder:

fb.run_multi(infiles, outfolder)

Plotting binning statistics 

After the binning has succeeded, you can generate a plot which shows the distribution of hits among the bins you defined. For this, call the following console command:

plot_binstats file_1_binned.h5 file_2_binned.h5 ... --save_as my_plotname.pdf

This will plot the statistics for the files file_1_binned.h5, file_2_binned.h5, … into the file my_plotname.pdf.

Using existing binnings 

You can use existing bin edges and mc info extractors from orcasong.bin_edges and orcasong.mc_info_extr. These were designed for specific detector layouts and productions, and might not work properly when used on other data.

Mode 2: Producing Graphs 

Generate the nodes of graphs from km3net data.

Basic Use 

Import the main class, the FileGraph (see orcasong.core.FileGraph), like this:

from orcasong.core import FileGraph

fg = FileGraph()

The FileGraph produces a list of nodes, each representing a hit. Since the number of hits varies from event to event, the hits of all events are saved in a long list (2d array), and a seperate datasets is saved that can be used to identify which hits belong to which events.

General usage 

Functionality that both modes have in common.

Calibration 

You can supply a detx file to the file binner, in order to calibrate the data on the fly:

fb = FileBinner(bin_edges_list, det_file="path/to/det_file.detx")

Adding mc_info 

Define a function my_extractor, which takes as an input a km3pipe blob, and outputs a dict mapping str to float. It should contain everything you need later down the pipeline, e.g. labels, event identifiers, …

This will be saved as a numpy structured array “y” in the output file, with the str being the dtype names. Set up like follows:

fb = FileBinner(bin_edges_list, extractor=my_extractor)