Skip to content

[WIP] Packs documentation #312

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
emedvedev merged 14 commits into from
Nov 28, 2016
Merged

[WIP] Packs documentation #312

Changes from 2 commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
3b5f757
Update pack docs
emedvedev Nov 11, 2016
c440961
Refinements
emedvedev Nov 11, 2016
878cd6a
Update upgrade notes.
Kami Nov 11, 2016
45c9713
Fix most of the contrib references
emedvedev Nov 11, 2016
18cc330
Fix @LindsayHill's comments
emedvedev Nov 11, 2016
d4176b0
Bring back dontrib reference and fix a typo
Nov 23, 2016
8057b10
Merge remote-tracking branch 'origin/master' into packs-docs
Nov 23, 2016
ec0265c
Fix the installation step for "working with packs"
emedvedev Nov 23, 2016
d85b184
Add the new page for migration notes and reference it in the "upgrade…
emedvedev Nov 23, 2016
2b14ecc
added orphan to reference/exchange so tests pass
LindsayHill Nov 23, 2016
020ddd1
Addressing comments
emedvedev Nov 28, 2016
c1b807b
ChatOps section
emedvedev Nov 28, 2016
cf43e79
Fix the new image links
emedvedev Nov 28, 2016
1880d85
Remove extra deploy reference
emedvedev Nov 28, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
196 changes: 132 additions & 64 deletions docs/source/packs.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,114 +6,182 @@ What is a Pack?
Pack is the unit of deployment for integrations and automations that extend |st2|. Typically a pack is organized along service or product boundaries e.g. AWS, Docker, Sensu etc. A pack can contain :doc:`Actions </actions>`, :doc:`Workflows </workflows>`,
:doc:`Rules </rules>`, :doc:`Sensors </sensors>`, :doc:`Aliases <chatops/aliases>`.

It is best to view a pack as the means to extend |st2| and allow it to integrate with an external systems. See `next section` to learn more about pack management.
It is best to view a pack as the means to extend |st2| and allow it to integrate with an external systems. Everything you create will also be part of a pack.

@LindsayHill LindsayHill Nov 11, 2016

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would say "...integrate with external systems."


Packs Location and Discovery
----------------------------
Managing Packs
--------------

.. note::

Everything about packs got better and easier in |st2| 2.1! Packs are naturally growing into a full package management solution inside |st2|. There are new API endpoints, CLI commands, repository structure for hosting packs, and a pack collection called StackStorm Exchange. Some of the older things are now deprecated. If you're using a previous version, it's time to upgrade or at least read the :doc:`upgrade notes </upgrade_notes>`.

@lakshmi-kannan lakshmi-kannan Nov 24, 2016

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

package management is incorrect. I'd skip that sentence entirely.


|st2| packs are managed through ``st2 pack ...`` commands: ``st2 pack -h`` will give you a useful overview if you just need a quick start.

A few packs (such as the ``core`` pack for basic StackStorm actions) come pre-installed with StackStorm. ``list`` and ``get`` are primary commands to work with local packs:

@LindsayHill LindsayHill Nov 11, 2016

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

slight change - "...list and get are the primary commands to work with local packs"


.. code-block:: bash

# List all installed packs
st2 pack list

When using |st2| and pack management actions, all the packs are by default installed into the
# Get detailed information about an installed pack
st2 pack get core

When using |st2| and pack management actions, all the packs are installed into the
system packs directory which defaults to ``/opt/stackstorm/packs``.

When |st2| searches for available packs it looks into the system packs directory and
into any additional directories which are specified in the ``packs_base_paths`` setting.
Discovering Packs
-----------------

To look for packs in additional directories, set the value of ``packs_base_paths`` in ``st2.conf``
(typically in :github_st2:`/etc/st2/st2.conf <conf/st2.prod.conf>`, as described in
:doc:`Configuration </install/config/config>`). The value must be a colon delimited string of directory paths.
There's over a hundred StackStorm packs already available to you! `StackStorm Exchange <https://exchange.stackstorm.org>`__ is a collection of ready-made packs submitted by StackStorm users and engineers, and there are packs for most of the popular cloud providers and DevOps tools.

For example:
In addition to the pack listing at `exchange.stackstorm.org <exchange.stackstorm.org>`__, you can search the pack index in CLI with ``st2 pack search`` and ``st2 pack show``:

::
.. code-block:: bash

[content]
packs_base_paths=/home/myuser1/packs:/home/myuser2/packs
# Search query is applied across all pack parameters.

Note: Directories are always searched from left to right in the order they are
specified, with the system packs directory always searched first.
# It will search through pack names:
st2 pack search sensu
# And keywords:
st2 pack search monitoring
# And description (use quotes for multi-word search):
st2 pack search "Amazon Web Services"
# And even pack author:
st2 pack search "Jon Middleton"

Getting a Pack
--------------
# Show an index entry for the pack
st2 pack show sensu

Pack management is done by |st2| actions from `packs` pack, pun intended. Run ``st2 action list --pack packs`` for a list of pack management actions.
Installing a Pack
-----------------

Some packs can be installed and run "as is" without any configurations.
Installing a pack is simple:

.. code-block:: bash

st2 run packs.install packs=docker,sensu repo_url=https://github.com/StackStorm/st2contrib.git
st2 pack install sensu

# You can also install multiple packs:
st2 pack install datadog github

This downloads the Sensu and Docker packs from the `StackStorm/st2contrib community repo on GitHub <https://github.com/StackStorm/st2contrib>`__, places them as local content under ``/opt/stackstorm/packs``, registers with |st2| and loads the content.
This command will download packs from the `StackStorm Exchange organization on GitHub <https://github.com/StackStorm-Exchange>`__, place them under ``/opt/stackstorm/packs``, and register them with |st2|.

By default packs are installed from the |st2| community repo. Use ``repo_url`` parameter to install a pack from a fork of `st2contrib <https://github.com/StackStorm/st2contrib>`__, or from a custom repo. The following example installs all the packs from `StackStorm/st2incubator <https://github.com/StackStorm/st2incubator>`__ - the repo where you find our experiments and work-in-progress:
Essentially, ``st2 pack install`` works with git repositories: there is one for every pack in the Exchange, and you can install your own packs from git just as easily.

.. code-block:: bash

st2 run packs.install packs=* register=all repo_url=https://github.com/StackStorm/st2incubator.git
st2 pack install https://github.com/emedvedev/chatops-training

If you are using a private repo to host your pack(s) on github, use:
``repo_url=https://username:password@github.com/username/repository.git``.
You can optionally pass ``subtree=True`` if the repo contains multiple packs within ``packs`` directory
of repo root just like `StackStorm/st2contrib community repo on GitHub <https://github.com/StackStorm/st2contrib>`_.
If the repo contains only one pack at the repo root, then use ``subtree=False``.
By default, the latest version of a pack will be installed, but you can specify a particular version, branch, tag, or even a commit hash.

To uninstall packs: ``st2 run packs.uninstall packs=docker,sensu``. This unloads and unregisters the content and deletes the packs from the disk.
.. code-block:: bash

The integration packs often require configurations to adjust to the environment. e.g. you will need to specify SMTP server for email, a puppet master URL for Puppet, or a Keystone endpoint and tenant credentials for OpenStack. The installation process is:
st2 pack install cloudflare=77ee04e
st2 pack install cloudflare=0.1.0
st2 pack install https://github.com/emedvedev/chatops-training=testing

@lakshmi-kannan lakshmi-kannan Nov 24, 2016

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing #. URL should be https://github.com/emedvedev/chatops-training#=testing

@enykeev enykeev Nov 24, 2016

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mmm. Why exactly? https://github.com/emedvedev/chatops-training=training is not a url, it's a [pack]=[version] with pack name as url and tag\branch as version. [pack]#[version] might have worked as url in some scenarios, but the team decided # is not a good delimiter for that purpose.


1. Download the pack with ``packs.download``
2. Check out the `README.md`. Adjust configurations per your environment. For
more information on pack configuration and how to configure the pack, please
refer to :doc:`/reference/pack_configs`.
3. Run pack setup via ``packs.setup_virtualenv``. It sets up a virtual environment and installs the dependencies listed in requirements.txt.
4. Load the pack into |st2| with ``pack.load register=all|actions|rules|sensors``.
Note that running ``st2 pack install`` on an installed pack will start an upgrade: your pack will be replaced with the version you're asking |st2| to install.

Let's install the Docker pack:
To uninstall a pack, use ``remove``:

.. code-block:: bash

# Download Docker pack from http://github.com/stackstorm/st2contrib
st2 run packs.download packs=docker
st2 pack remove sensu

Configuring a Pack
------------------

# Set up a virtual environment for this pack and install all the pack dependencies
# listed in requirements.txt (if any).
# Virtual environment provides isolated Python environment for sensors and Python runner
# actions.
st2 run packs.setup_virtualenv packs=docker
Integration packs often require configuration to adjust to the environment. e.g. you will need to specify SMTP server for email, a puppet master URL for Puppet, or a Keystone endpoint and tenant credentials for OpenStack.

# Check out README.md and if necessary, adjust configuration for your environment
less /opt/stackstorm/packs/docker/README.md
Most packs that require configuration can be configured interactively:

# Load ALL the content: actions, sensors, rules
# If you don't want to load sample rules by default, do
# st2 run packs.load register=sensors && st2 run packs.load register=actions
st2 run packs.load register=all
.. code-block:: bash

st2 pack config cloudflare

# To pick up sensors, we need to bounce the st2sensorcontainer process.
# Note: live update coming soon and this won't be needed.
st2 run packs.restart_component servicename=st2sensorcontainer
You will be prompted for configuration parameters in a nice interactive tool with descriptions, suggestions, and defaults. You will also be asked to verify your final config file in a text editor before saving it: it's optional, and most packs don't require more than two or three fields, but we have to comply with Health and Safety.

# Verify that the Docker pack was installed
st2 action list --pack=docker
st2 sensor list --pack=docker
st2 trigger list --pack=docker
Some packs will require old-school manual configuration: in that case, you will have to edit a ``config.yaml`` file inside the ``/opt/stackstorm/packs/<pack>`` directory and reload the pack with ``st2 pack register <pack>``. Your config will not be overwritten on pack upgrades, and to be extra safe, you can save it as ``/opt/stackstorm/configs/<pack>.yaml`` to keep it away from the pack source, which is something we recommend.

The Docker pack is now installed and ready to use.
Packs may contain automation rules. Rules are not loaded by default, because you may want to review and adjust them before loading. Use ``st2 pack register`` to register rules in the pack.

@lakshmi-kannan lakshmi-kannan Nov 24, 2016

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

st2ctl now loads rules by default. Why is it different here?


Packs may contain automations - rules and workflows. Rules are not loaded by default - you may want to review and adjust them before loading. Pass ``register=all`` option to ``packs.install`` and ``packs.load`` actions to get the rules loaded. Use ``st2ctl reload`` for fine control - ``packs.load`` is an st2 action wrapper around it.
.. code-block:: bash

.. note:: Pack management is implemented as a pack of st2 actions. Check out :github_st2:`/opt/stackstorm/packs </contrib/packs>` for examples of defining actions and workflows.
st2 pack register sensu --types=rule

Creating a Pack
---------------
Developing a Pack
-----------------

See :doc:`/reference/packs` for details on how to package your integrations and automations in a pack, how to publish it, and how to contribute it to the |st2| community.
See :doc:`/reference/packs` for details on how to package your integrations and automations in a pack, how to fork a pack for development or create your own, how to publish it, and how to contribute it to the |st2| community. If you're planning to develop any |st2| content, we would strongly suggest getting yourself familiar with that page: every piece of content in StackStorm has to belong to a pack, and a good understanding of pack workflow will make your development process much easier!

.. rubric:: What's Next?

* Explore existing packs for many common products and tools from `StackStorm community <http://www.stackstorm.com/community/>`__ - `st2contrib <https://github.com/StackStorm/st2contrib>`__.
* Explore existing packs for many common products and tools: `StackStorm Exchange <https://exchange.stackstorm.org>`__.
* Learn how to write a pack and contribute to the community - :doc:`/reference/packs`.
* Learn how to write :ref:`custom sensors <ref-sensors-authoring-a-sensor>` and :ref:`custom actions <ref-actions-writing-custom>`.
* Check out `tutorials on stackstorm.com <http://stackstorm.com/category/tutorials/>`__ - a growing set of practical examples of automating with |st2|.
* For information on pack testing, please see the :doc:`Pack Testing </development/pack_testing>` page.

Advanced Topics
---------------

Installing packs from private repositories
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

If you're installing a pack from a private repository on GitHub, you can use https auth with a `personal access token <https://help.github.com/articles/creating-an-access-token-for-command-line-use/>`__, or create a `deploy key <https://github.com/blog/2024-read-only-deploy-keys>`__ to use SSH.

Access tokens are used with HTTPS auth: ``st2 pack install https://<user>:<token>@github.com/username/repo.git``.

Deploy keys are used with ``git@`` urls, and require the system user running the command (stanley or root, depending on your configuration) to have a private key, but they are more secure and can be configured on the per-repo basis.

Other git hosting services should also support either SSH or HTTPS auth, and would be configured in a similar fashion.

Pack location
~~~~~~~~~~~~~

When |st2| searches for available packs, it looks into the system packs directory and
into any additional directories which are specified in the ``packs_base_paths`` setting.

To look for packs in additional directories, set the value of ``packs_base_paths`` in ``st2.conf``
(typically in :github_st2:`/etc/st2/st2.conf <conf/st2.prod.conf>`, as described in
:doc:`Configuration </install/config/config>`). The value must be a colon delimited string of directory paths.

For example:

::

[content]
packs_base_paths=/home/myuser1/packs:/home/myuser2/packs

Note: Directories are always searched from left to right in the order they are
specified, with the system packs directory always searched first.

Working with pack indexes
~~~~~~~~~~~~~~~~~~~~~~~~~

When you run pack management commands like ``st2 pack install sensu`` or ``st2 pack search git``, |st2| uses a pack index file to resolve short names and perform searches. A pack index is, essentially, a JSON object: it contains metadata and URLs for all available packs.

`The StackStorm Exchange index file <https://index.stackstorm.org/v1/index.json>`__ is a default file used by all |st2| instances, and a good example of the index format. The file is hosted on GitHub (`StackStorm-Exchange/index <https://github.com/stackstorm-exchange/index>`__) and proxied through CloudFlare CDN.

The index path is specified in ``st2.conf`` as ``content.index_url``. You can replace the default index, or even use more than one with a comma-separated list:

::
[content]
index_url=https://my-super-index.org/index.json,https://exchange.stackstorm.org/v1/index.json

Contents from all specified indexes will merge, priority arranged left to right. In the example above, ``sensu`` pack in your own index would override ``sensu`` pack from the Exchange.

There are multiple reasons to consider hosting your own index, especially with HA deployments or multi-server setups:

* mirroring: in case the main index is not available, your mirror will be used;
* forking: if you fork Exchange packs, you can create an index that is going to use your forks;
* enterprise restrictions: if you need pack names to resolve, but can't install from github, you can specify your own index as the only source;
* a centralized resolver: in a multi-server deployment, you can host an index to keep repo URLs in a centralized location;
* bragging rights: get your own packs resolvable by short names because the cool kids are doing it.

In most cases there are many other ways to solve the same problem, but sometimes a pack index is a viable alternative. Create your index file and make it accessible over HTTPS, change the config—and you're good!

When you have to monitor index health, the ``/packs/index/health`` API endpoint will show you the state of all indexes used by your |st2| instance.

.. include:: /__engage_community.rst