
Formulae should not depend on both OpenSSL and LibreSSL (even optionally). This is to avoid descending into madness where every formulae that could use LibreSSL has to have option and switching logic. Homebrew has standardised on OpenSSL and will do so everywhere that LibreSSL is not a hard requirement.
Homebrew's Formula API
This is the (partially) documented public API for Homebrew.
The main class you should look at is the {Formula} class (and classes linked from there). That's the class that's used to create Homebrew formulae (i.e. package descriptions). Assume anything else you stumble upon is private.
You may also find the Formula Cookbook and Ruby Style Guide helpful in creating formulae.
Good luck!