fireworks.core package¶
Subpackages¶
Submodules¶
fireworks.core.firework module¶
-
class
fireworks.core.firework.
FWAction
(stored_data=None, exit=False, update_spec=None, mod_spec=None, additions=None, detours=None, defuse_children=False, defuse_workflow=False)¶ Bases:
fireworks.utilities.fw_serializers.FWSerializable
A FWAction encapsulates the output of a Firetask (it is returned by a Firetask after the Firetask completes). The FWAction allows a user to store rudimentary output data as well as return commands that alter the workflow.
-
classmethod
from_dict
(*args, **kwargs)¶
-
skip_remaining_tasks
¶ If the FWAction gives any dynamic action, we skip the subsequent Firetasks
Returns: bool
-
to_dict
(*args, **kwargs)¶
-
classmethod
-
class
fireworks.core.firework.
FireTaskBase
(*args, **kwargs)¶
-
class
fireworks.core.firework.
FiretaskBase
(*args, **kwargs)¶ Bases:
collections.defaultdict
,fireworks.utilities.fw_serializers.FWSerializable
FiretaskBase is used like an abstract class that defines a computing task (Firetask). All Firetasks should inherit from FiretaskBase.
You can set parameters of a Firetask like you’d use a dict.
-
classmethod
from_dict
(*args, **kwargs)¶
-
required_params
= []¶
-
run_task
(fw_spec)¶ This method gets called when the Firetask is run. It can take in a Firework spec, perform some task using that data, and then return an output in the form of a FWAction.
Parameters: fw_spec (dict) – A Firework spec. This comes from the master spec. In addition, this spec contains a special “_fw_env” key that contains the env settings of the FWorker calling this method. This provides for abstracting out certain commands or settings. For example, “foo” may be named “foo1” in resource 1 and “foo2” in resource 2. The FWorker env can specify { “foo”: “foo1”}, which maps an abstract variable “foo” to the relevant “foo1” or “foo2”. You can then write a task that uses fw_spec[“_fw_env”][“foo”] that will work across all these multiple resources. Returns: (FWAction)
-
to_dict
(*args, **kwargs)¶
-
classmethod
-
class
fireworks.core.firework.
FiretaskMeta
¶ Bases:
abc.ABCMeta
-
class
fireworks.core.firework.
Firework
(tasks, spec=None, name=None, launches=None, archived_launches=None, state=u'WAITING', created_on=None, fw_id=None, parents=None, updated_on=None)¶ Bases:
fireworks.utilities.fw_serializers.FWSerializable
A Firework is a workflow step and might be contain several Firetasks.
-
STATE_RANKS
= {u'PAUSED': 0, u'RUNNING': 4, u'ARCHIVED': -2, u'WAITING': 1, u'FIZZLED': -1, u'READY': 2, u'RESERVED': 3, u'COMPLETED': 5, u'DEFUSED': 0}¶
-
classmethod
from_dict
(*args, **kwargs)¶
-
to_db_dict
()¶ Return firework dict with updated launches and state.
-
to_dict
(*args, **kwargs)¶
-
-
class
fireworks.core.firework.
Launch
(state, launch_dir, fworker=None, host=None, ip=None, trackers=None, action=None, state_history=None, launch_id=None, fw_id=None)¶ Bases:
fireworks.utilities.fw_serializers.FWSerializable
,object
A Launch encapsulates data about a specific run of a Firework on a computing resource.
-
classmethod
from_dict
(*args, **kwargs)¶
-
last_pinged
¶ **Returns* – datetime* – the time the Launch last pinged a heartbeat that it was still running
-
reservedtime_secs
¶ **Returns* – int* – number of seconds the Launch was stuck as RESERVED in a queue.
-
set_reservation_id
(reservation_id)¶ Adds the job_id to the reservation.
Parameters: reservation_id (int or str) – the id of the reservation (e.g., queue reservation)
-
to_db_dict
(*args, **kwargs)¶
-
to_dict
(*args, **kwargs)¶
-
classmethod
-
class
fireworks.core.firework.
Tracker
(filename, nlines=25, content=u'', allow_zipped=False)¶ Bases:
fireworks.utilities.fw_serializers.FWSerializable
,object
A Tracker monitors a file and returns the last N lines for updating the Launch object.
-
MAX_TRACKER_LINES
= 1000¶
-
classmethod
from_dict
(m_dict)¶
-
to_dict
()¶
-
-
class
fireworks.core.firework.
Workflow
(fireworks, links_dict=None, name=None, metadata=None, created_on=None, updated_on=None, fw_states=None)¶ Bases:
fireworks.utilities.fw_serializers.FWSerializable
A Workflow connects a group of FireWorks in an execution order.
-
class
Links
(*args, **kwargs)¶ Bases:
dict
,fireworks.utilities.fw_serializers.FWSerializable
An inner class for storing the DAG links between FireWorks
-
classmethod
from_dict
(m_dict)¶
-
nodes
¶ Return list of all nodes
-
parent_links
¶ Return a dict of child and its parents.
Note: if performance of parent_links becomes an issue, override delitem/setitem to update parent_links
-
to_db_dict
()¶ Convert to str form for Mongo, which cannot have int keys .
Returns: dict
-
to_dict
()¶ Convert to str form for Mongo, which cannot have int keys.
Returns: dict
-
classmethod
-
Workflow.
append_wf
(new_wf, fw_ids, detour=False, pull_spec_mods=False)¶ Method to add a workflow as a child to a Firework Note: detours must have children that have STATE_RANK that is WAITING or below
Parameters: Returns: list of Firework ids that were updated or new
Return type: [int]
-
Workflow.
apply_action
(action, fw_id)¶ Apply a FWAction on a Firework in the Workflow.
Parameters: Returns: list of Firework ids that were updated or new
Return type: [int]
-
classmethod
Workflow.
from_Firework
(fw, name=None, metadata=None)¶ Return Workflow from the given Firework.
Parameters: Returns: Workflow
-
classmethod
Workflow.
from_dict
(m_dict)¶ Return Workflow from its dict representation.
Parameters: m_dict (dict) – either a Workflow dict or a Firework dict Returns: Workflow
-
classmethod
Workflow.
from_wflow
(wflow)¶ Create a fresh Workflow from an existing one.
Parameters: wflow (Workflow) – Returns: Workflow
-
Workflow.
fws
¶ Return list of all fireworks
-
Workflow.
leaf_fw_ids
¶ Gets leaf FireWorks of this workflow (those with no children).
Returns: Firework ids of leaf FWs Return type: [int]
-
Workflow.
refresh
(fw_id, updated_ids=None)¶ Refreshes the state of a Firework and any affected children.
Parameters: Returns: list of Firework ids that were updated
Return type:
-
Workflow.
rerun_fw
(fw_id, updated_ids=None)¶ Archives the launches of a Firework so that it can be re-run.
Parameters: Returns: list of Firework ids that were updated
Return type: [int]
-
Workflow.
reset
(reset_ids=True)¶ Reset the states of all Fireworks in this workflow to ‘WAITING’.
Parameters: reset_ids (bool) – if True
, give each Firework a new id.
-
Workflow.
root_fw_ids
¶ Gets root FireWorks of this workflow (those with no parents).
Returns: Firework ids of root FWs Return type: [int]
-
Workflow.
to_db_dict
()¶
-
Workflow.
to_dict
()¶
-
Workflow.
to_display_dict
()¶
-
class
fireworks.core.fworker module¶
-
class
fireworks.core.fworker.
FWorker
(name=u'Automatically generated Worker', category=u'', query=None, env=None)¶ Bases:
fireworks.utilities.fw_serializers.FWSerializable
-
classmethod
auto_load
()¶ Returns FWorker object from settings file(my_fworker.yaml).
-
classmethod
from_dict
(*args, **kwargs)¶
-
query
¶ Returns updated query dict.
-
to_dict
(*args, **kwargs)¶
-
classmethod
fireworks.core.launchpad module¶
-
class
fireworks.core.launchpad.
LaunchPad
(host=u'localhost', port=27017, name=u'fireworks', username=None, password=None, logdir=None, strm_lvl=None, user_indices=None, wf_user_indices=None, ssl_ca_file=None)¶ Bases:
fireworks.utilities.fw_serializers.FWSerializable
The LaunchPad manages the FireWorks database.
-
add_offline_run
(launch_id, fw_id, name)¶ Add the launch and firework to the offline_run collection.
Parameters:
-
add_wf
(wf, reassign_all=True)¶ Add workflow(or firework) to the launchpad. The firework ids will be reassigned.
Parameters: wf (Workflow/Firework) – Returns: mapping between old and new Firework ids Return type: dict
-
append_wf
(new_wf, fw_ids, detour=False, pull_spec_mods=True)¶ Append a new workflow on top of an existing workflow.
Parameters: - new_wf (Workflow) – The new workflow to append
- fw_ids ([int]) – The parent fw_ids at which to append the workflow
- detour (bool) – Whether to connect the new Workflow in a “detour” style, i.e., move
- children of the parent fw_ids to the new_wf (original) –
- pull_spec_mods (bool) – Whether the new Workflow should pull the FWActions of the parent fw_ids
-
archive_wf
(fw_id)¶ Archive the workflow containing the given firework id.
Parameters: fw_id (int) – firework id
-
classmethod
auto_load
()¶
-
cancel_reservation
(launch_id)¶ given the launch id, cancel the reservation and rerun the fireworks
-
cancel_reservation_by_reservation_id
(reservation_id)¶ Given the reservation id, cancel the reservation and rerun the corresponding fireworks.
-
change_launch_dir
(launch_id, launch_dir)¶ Change the launch directory corresponding to the given launch id.
Parameters:
-
checkout_fw
(fworker, launch_dir, fw_id=None, host=None, ip=None, state=u'RUNNING')¶ Checkout the next ready firework, mark it with the given state(RESERVED or RUNNING) and return it to the caller. The caller is responsible for running the Firework.
Parameters: - fworker (FWorker) – A FWorker instance
- launch_dir (str) – the dir the FW will be run in (for creating a Launch object)
- fw_id (int) – Firework id
- host (str) – the host making the request (for creating a Launch object)
- ip (str) – the ip making the request (for creating a Launch object)
- state (str) – RESERVED or RUNNING, the fetched firework’s state will be set to this value.
Returns: firework and the new launch id
Return type:
-
complete_launch
(launch_id, action=None, state=u'COMPLETED')¶ Internal method used to mark a Firework’s Launch as completed.
Parameters: Returns: updated launch
Return type:
-
defuse_fw
(fw_id, rerun_duplicates=True)¶ Given the firework id, defuse the firework and refresh the workflow.
Parameters:
-
defuse_wf
(fw_id, defuse_all_states=True)¶ Defuse the workflow containing the given firework id.
Parameters:
-
delete_wf
(fw_id)¶ Delete the workflow containing firework with the given id.
Parameters: fw_id (int) – Firework id
-
detect_lostruns
(expiration_secs=14400, fizzle=False, rerun=False, max_runtime=None, min_runtime=None, refresh=False)¶ Detect lost runs i.e running fireworks that haven’t been updated within the specified time limit or running firework whose launch has been marked fizzed or completed.
Parameters: - expiration_secs (seconds) – expiration time in seconds
- fizzle (bool) – if True, mark the lost runs fizzed
- rerun (bool) – if True, mark the lost runs fizzed and rerun
- max_runtime (seconds) – maximum run time
- min_runtime (seconds) – minimum run time
- refresh (bool) – if True, refresh the workflow with inconsistent fireworks.
Returns: - tuple of list of lost launch ids, lost firework ids and
inconsistent firework ids.
Return type:
-
detect_unreserved
(expiration_secs=1209600, rerun=False)¶ Return the reserved launch ids that have not been updated for a while.
Parameters: - expiration_secs (seconds) – time limit
- rerun (bool) – if True, the expired reservations are cancelled and the fireworks rerun.
Returns: list of expired lacunh ids
Return type: [int]
-
forget_offline
(launchid_or_fwid, launch_mode=True)¶ Unmark the offline run for the given launch or firework id.
Parameters:
-
classmethod
from_dict
(d)¶
-
get_fw_by_id
(fw_id)¶ Given a Firework id, give back a Firework object.
Parameters: fw_id (int) – Firework id. Returns: Firework object
-
get_fw_dict_by_id
(fw_id)¶ Given firework id, return firework dict.
Parameters: fw_id (int) – firework id Returns: dict
-
get_fw_ids
(query=None, sort=None, limit=0, count_only=False, launches_mode=False)¶ Return all the fw ids that match a query.
Parameters: Returns: list of firework ids matching the query
Return type:
-
get_fw_ids_from_reservation_id
(reservation_id)¶ Given the reservation id, return the list of firework ids.
Parameters: reservation_id (int) – Returns: list of firework ids. Return type: [int]
-
get_launch_by_id
(launch_id)¶ Given a Launch id, return details of the Launch.
Parameters: launch_id (int) – launch id Returns: Launch object
-
get_logdir
()¶ Return the log directory.
AJ: This is needed for job packing due to Proxy objects not being fully featured...
-
get_new_fw_id
()¶ Checkout the next Firework id
-
get_new_launch_id
()¶ Checkout the next Launch id
-
get_reservation_id_from_fw_id
(fw_id)¶ Given the firework id, return the reservation id
-
get_tracker_data
(fw_id)¶ Parameters: fw_id (id) – firework id Returns: list tracker dicts Return type: [dict]
-
get_wf_by_fw_id
(fw_id)¶ Given a Firework id, give back the Workflow containing that Firework.
Parameters: fw_id (int) – Returns: A Workflow object
-
get_wf_by_fw_id_lzyfw
(fw_id)¶ Given a FireWork id, give back the Workflow containing that FireWork.
Parameters: fw_id (int) – Returns: A Workflow object
-
get_wf_ids
(query=None, sort=None, limit=0, count_only=False)¶ Return one fw id for all workflows that match a query.
Parameters: Returns: list of firework ids
Return type:
-
get_wf_summary_dict
(fw_id, mode=u'more')¶ A much faster way to get summary information about a Workflow by querying only for needed information.
Parameters: Returns: information about Workflow.
Return type:
-
maintain
(infinite=True, maintain_interval=None)¶ Perform launchpad maintenance: detect lost runs and unreserved RESERVE launches.
Parameters: - infinite (bool) –
- maintain_interval (seconds) – sleep time
-
mark_fizzled
(launch_id)¶ Mark the launch corresponding to the given id as FIZZLED.
Parameters: launch_id (int) – launch id Returns: updated launch Return type: dict
-
pause_fw
(fw_id)¶ Given the firework id, pauses the firework and refresh the workflow
Parameters: fw_id (int) – firework id
-
pause_wf
(fw_id)¶ Pause the workflow containing the given firework id.
Parameters:
-
ping_launch
(launch_id, ptime=None)¶ Ping that a Launch is still alive: updates the ‘update_on ‘field of the state history of a Launch.
Parameters:
-
recover_offline
(launch_id, ignore_errors=False, print_errors=False)¶ Update the launch state using the offline data in FW_offline.json file.
Parameters: Returns: firework id if the recovering fails otherwise None
-
reignite_fw
(fw_id)¶ Given the firework id, re-ignite(set state=WAITING) the defused firework.
Parameters: fw_id (int) – firework id
-
reignite_wf
(fw_id)¶ Reignite the workflow containing the given firework id.
Parameters: fw_id (int) – firework id
-
rerun_fw
(fw_id, rerun_duplicates=True, clear_recovery=False)¶ Rerun the firework corresponding to the given id.
Parameters: Returns: list of firework ids that were rerun
Return type: [int]
-
rerun_fws_task_level
(fw_id, rerun_duplicates=True, launch_id=None, recover_mode=None)¶ Rerun a fw at the task level.
Parameters: - fw_id (int) – fw_id to rerun
- rerun_duplicates (bool) – also rerun duplicate FWs
- launch_id (int) – launch id to rerun, if known. otherwise the last launch_id will be used
- recover_mode (str) – use “prev_dir” to run again in previous dir, “cp” to try to copy data to new dir, or None to start from scratch
Returns: list of rerun firework ids.
Return type: [int]
-
reserve_fw
(fworker, launch_dir, host=None, ip=None, fw_id=None)¶ Checkout the next ready firework and mark the launch reserved.
Parameters: Returns: the checked out firework and the new launch id
Return type:
-
reset
(password, require_password=True, max_reset_wo_password=25)¶ Create a new FireWorks database. This will overwrite the existing FireWorks database! To safeguard against accidentally erasing an existing database, a password must be entered.
Parameters: - password (str) – A String representing today’s date, e.g. ‘2012-12-31’
- require_password (bool) – Whether a password is required to reset the DB. Setting to false is dangerous because running code unintentionally could clear your DB - use max_reset_wo_password to minimize risk.
- max_reset_wo_password (int) – A failsafe; when require_password is set to False, FWS will not clear DBs that contain more workflows than this parameter
-
restore_backup_data
(launch_id, fw_id)¶ For the given launch id and firework id, restore the back up data.
-
resume_fw
(fw_id)¶ Given the firework id, resume (set state=WAITING) the paused firework.
Parameters: fw_id (int) – firework id
-
run_exists
(fworker=None)¶ Checks to see if the database contains any FireWorks that are ready to run.
Returns: True if the database contains any FireWorks that are ready to run. Return type: bool
-
set_priority
(fw_id, priority)¶ Set priority to the firework with the given id.
Parameters: - fw_id (int) – firework id
- priority –
-
set_reservation_id
(launch_id, reservation_id)¶ Set reservation id to the launch corresponding to the given launch id.
Parameters:
-
to_dict
()¶ Note: usernames/passwords are exported as unencrypted Strings!
-
tuneup
(bkground=True)¶ Database tuneup: build indexes
-
update_spec
(fw_ids, spec_document, mongo=False)¶ Update fireworks with a spec. Sometimes you need to modify a firework in progress.
Parameters: - [int] (fw_ids) – All fw_ids to modify.
- spec_document (dict) – The spec document. Note that only modifications to the spec key are allowed. So if you supply {“_tasks.1.parameter”: “hello”}, you are effectively modifying spec._tasks.1.parameter in the actual fireworks collection.
- mongo (bool) – spec_document uses mongo syntax to directly update the spec
-
-
class
fireworks.core.launchpad.
LazyFirework
(fw_id, fw_coll, launch_coll)¶ Bases:
object
A LazyFirework only has the fw_id, and grabs other data just-in-time. This representation can speed up Workflow loading as only “important” FWs need to be fully loaded.
-
archived_launches
¶
-
created_on
¶
-
db_fields
= (u'name', u'fw_id', u'spec', u'created_on', u'state')¶
-
db_launch_fields
= (u'launches', u'archived_launches')¶
-
full_fw
¶
-
launches
¶
-
name
¶
-
parents
¶
-
partial_fw
¶
-
spec
¶
-
state
¶
-
tasks
¶
-
to_db_dict
()¶
-
to_dict
()¶
-
updated_on
¶
-
-
exception
fireworks.core.launchpad.
LockedWorkflowError
¶ Bases:
exceptions.ValueError
Error raised if the context manager WFLock can’t acquire the lock on the WF within the selected time interval (WFLOCK_EXPIRATION_SECS), if the killing of the lock is disabled (WFLOCK_EXPIRATION_KILL)
-
class
fireworks.core.launchpad.
WFLock
(lp, fw_id, expire_secs=300, kill=False)¶ Bases:
object
Lock a Workflow, i.e. for performing update operations Raises a LockedWorkflowError if the lock couldn’t be acquired withing expire_secs and kill==False. Calling functions are responsible for handling the error in order to avoid database inconsistencies.
fireworks.core.rocket module¶
-
class
fireworks.core.rocket.
Rocket
(launchpad, fworker, fw_id)¶ The Rocket fetches a workflow step from the FireWorks database and executes it.
-
decorate_fwaction
(fwaction, my_spec, m_fw, launch_dir)¶
-
run
()¶ Run the rocket (check out a job from the database and execute it)
-
-
fireworks.core.rocket.
background_task
(btask, spec, stop_event, master_thread)¶
-
fireworks.core.rocket.
do_ping
(launchpad, launch_id)¶
-
fireworks.core.rocket.
ping_launch
(launchpad, launch_id, stop_event, master_thread)¶
-
fireworks.core.rocket.
start_background_task
(btask, spec)¶
-
fireworks.core.rocket.
start_ping_launch
(launchpad, launch_id)¶
-
fireworks.core.rocket.
stop_backgrounds
(ping_stop, btask_stops)¶
fireworks.core.rocket_launcher module¶
-
fireworks.core.rocket_launcher.
get_fworker
(fworker)¶
-
fireworks.core.rocket_launcher.
launch_rocket
(launchpad, fworker=None, fw_id=None, strm_lvl=u'INFO')¶ Run a single rocket in the current directory.
Parameters: Returns: bool
-
fireworks.core.rocket_launcher.
rapidfire
(launchpad, fworker=None, m_dir=None, nlaunches=0, max_loops=-1, sleep_time=None, strm_lvl=u'INFO', timeout=None)¶ Keeps running Rockets in m_dir until we reach an error. Automatically creates subdirectories for each Rocket. Usually stops when we run out of FireWorks from the LaunchPad.
Parameters: - launchpad (LaunchPad) –
- fworker (FWorker object) –
- m_dir (str) – the directory in which to loop Rocket running
- nlaunches (int) – 0 means ‘until completion’, -1 or “infinite” means to loop until max_loops
- max_loops (int) – maximum number of loops (default -1 is infinite)
- sleep_time (int) – secs to sleep between rapidfire loop iterations
- strm_lvl (str) – level at which to output logs to stdout
- timeout (int) – of seconds after which to stop the rapidfire process