defmodule RustlerPrecompiled do @moduledoc """ Download and use precompiled NIFs safely with checksums. Rustler Precompiled is a tool for library maintainers that rely on Rustler. It helps by removing the need to have the Rust compiler installed in the user's machine. Check the [Precompilation Guide](PRECOMPILATION_GUIDE.md) for details. ## Example defmodule MyApp.MyNative do use RustlerPrecompiled, otp_app: :my_app, crate: "my_app_nif", base_url: "https://github.com/me/my_project/releases/download/v0.1.0", version: "0.1.0" end ## Options * `:otp_app` - The OTP app name that the dynamic library will be loaded from. * `:crate` - The name of Rust crate if different from the `:otp_app`. This is optional. * `:base_url` - Location where to find the NIFs from. This should be one of the following: * A URL to a directory containing the NIFs. The name of the NIF will be appended to it and a GET request will be made. Works well with public GitHub releases. * A tuple of `{URL, headers}`. The headers should be a list of key-value pairs. This is useful when the NIFs are hosted in a private server. * A tuple of `{module, function}` where the `function` is an atom representing the function name in that module. It's expected a function of arity 1, where the NIF file name is given, and it should return a URL or a tuple of `{URL, headers}`. This should be used for all cases not covered by the above. For example when multiple requests have to be made, like when using a private GitHub release through the GitHub API, or when the URLs don't resemble a simple directory. * `:version` - The version of precompiled assets (it is part of the NIF filename). * `:force_build` - Force the build with `Rustler`. This is `false` by default, but if your `:version` is a pre-release (like "2.1.0-dev"), this option will always be set `true`. You can also configure this option by setting an application env like this: config :rustler_precompiled, :force_build, your_otp_app: true It is important to add the ":rustler" package to your dependencies in order to force the build. To do that, just add it to your `mix.exs` file: {:rustler, ">= 0.0.0", optional: true} In case you want to force the build for all packages using RustlerPrecompiled, you can set the application config `:force_build_all`, or the env var `RUSTLER_PRECOMPILED_FORCE_BUILD_ALL` (see details below): config :rustler_precompiled, force_build_all: true * `:targets` - A list of targets [supported by Rust](https://doc.rust-lang.org/rustc/platform-support.html) for which precompiled assets are available. By default the following targets are configured: #{Enum.map_join(RustlerPrecompiled.Config.default_targets(), "\n", &" - `#{&1}`")} * `:nif_versions` - A list of OTP NIF versions for which precompiled assets are available. A NIF version is usually compatible with two OTP minor versions, and an older NIF is usually compatible with newer OTPs. The available versions are the following: * `2.14` - for OTP 21 and above. * `2.15` - for OTP 22 and above. * `2.16` - for OTP 24 and above. * `2.17` - for OTP 26 and above. By default the following NIF versions are configured: #{Enum.map_join(RustlerPrecompiled.Config.default_nif_versions(), "\n", &" - `#{&1}`")} Check the compatibiliy table between Elixir and OTP in: https://hexdocs.pm/elixir/compatibility-and-deprecations.html#compatibility-between-elixir-and-erlang-otp * `:max_retries` - The maximum of retries before giving up. Defaults to `3`. Retries can be disabled with `0`. * `:variants` - A map with alternative versions of a given target. This is useful to support specific versions of dependencies, such as an old glibc version, or to support restrict CPU features, like AVX on x86_64. The order of variants matters, because the first one that returns `true` is going to be selected. Example: %{"x86_64-unknown-linux-gnu" => [old_glibc: fn _config -> has_old_glibc?() end]} In case "force build" is used, all options except the ones use by RustlerPrecompiled are going to be passed down to `Rustler`. So if you need to configure the build, check the `Rustler` options. ## Environment variables This project reads some system environment variables. They are all optional, but they can change the behaviour of this library at **compile time** of your project. They are: * `HTTP_PROXY` or `http_proxy` - Sets the HTTP proxy configuration. * `HTTPS_PROXY` or `https_proxy` - Sets the HTTPS proxy configuration. * `HEX_CACERTS_PATH` - Sets the path for a custom CA certificates file. If unset, defaults to `CAStore.file_path/0`. * `MIX_XDG` - If present, sets the OS as `:linux` for the `:filename.basedir/3` when getting an user cache dir. * `TARGET_ARCH` - The CPU target architecture. This is useful for when building your Nerves project, where your host CPU is different from your target CPU. Note that Nerves sets this value automatically when building your project. Examples: `arm`, `aarch64`, `x86_64`, `riscv64`. * `TARGET_ABI` - The target ABI (e.g., `gnueabihf`, `musl`). This is set by Nerves as well. * `TARGET_VENDOR` - The target vendor (e.g., `unknown`, `apple`, `pc`). This is **not** set by Nerves. If any of the `TARGET_` env vars is set, but `TARGET_VENDOR` is empty, then we change the target vendor to `unknown` that is the default value for Linux systems. * `TARGET_OS` - The target operational system. This is always `linux` for Nerves. * `RUSTLER_PRECOMPILED_GLOBAL_CACHE_PATH` - The global cache path directory. If set, it will ignore the default cache path resolution, thus ignoring `MIX_XDG`, and will try to fetch the artifacts from that path. In case the desired artifact is not found, a download is going to start. This variable is important for systems that cannot perform a download at compile time, like inside NixOS. It will require people to previously download the artifacts to that path. * `RUSTLER_PRECOMPILED_FORCE_BUILD_ALL` - If set to "1" or "true", it will override the `:force_build` configuration for all packages, and will force the build for them all. You can set the `:force_build_all` configuration to `true` to have the same effect. Note that all packages using `RustlerPrecompiled` will be affected by these environment variables. For more details about Nerves env vars, see https://hexdocs.pm/nerves/environment-variables.html """ defmacro __using__(opts) do force = if Code.ensure_loaded?(Rustler) do quote do use Rustler, only_rustler_opts end else quote do raise "Rustler dependency is needed to force the build. " <> "Add it to your `mix.exs` file: `{:rustler, \">= 0.0.0\", optional: true}`" end end quote do require Logger opts = unquote(opts) otp_app = Keyword.fetch!(opts, :otp_app) opts = if Application.compile_env( :rustler_precompiled, :force_build_all, System.get_env("RUSTLER_PRECOMPILED_FORCE_BUILD_ALL") in ["1", "true"] ) do Keyword.put(opts, :force_build, true) else Keyword.put_new( opts, :force_build, Application.compile_env(:rustler_precompiled, [:force_build, otp_app]) ) end case RustlerPrecompiled.__using__(__MODULE__, opts) do {:force_build, only_rustler_opts} -> unquote(force) {:ok, config} -> @on_load :load_rustler_precompiled @rustler_precompiled_load_from config.load_from @rustler_precompiled_load_data config.load_data @doc false def load_rustler_precompiled do # Remove any old modules that may be loaded so we don't get # {:error, {:upgrade, 'Upgrade not supported by this NIF library.'}} :code.purge(__MODULE__) {otp_app, path} = @rustler_precompiled_load_from load_path = otp_app |> Application.app_dir(path) |> to_charlist() :erlang.load_nif(load_path, @rustler_precompiled_load_data) end {:error, precomp_error} -> raise precomp_error end end end # A helper function to extract the logic from __using__ macro. @doc false def __using__(module, opts) do config = opts |> Keyword.put_new(:module, module) |> RustlerPrecompiled.Config.new() case build_metadata(config) do {:ok, metadata} -> # We need to write metadata in order to run Mix tasks. with {:error, error} <- write_metadata(module, metadata) do require Logger Logger.warning( "Cannot write metadata file for module #{inspect(module)}. Reason: #{inspect(error)}. " <> "This is only an issue if you need to use the rustler_precompiled mix tasks for publishing a package." ) end if config.force_build? do rustler_opts = Keyword.drop(opts, [ :base_url, :version, :force_build, :targets, :nif_versions, :max_retries, :variants ]) {:force_build, rustler_opts} else with {:error, precomp_error} <- RustlerPrecompiled.download_or_reuse_nif_file(config, metadata) do message = """ Error while downloading precompiled NIF: #{precomp_error}. You can force the project to build from scratch with: config :rustler_precompiled, :force_build, #{config.otp_app}: true In order to force the build, you also need to add Rustler as a dependency in your `mix.exs`: {:rustler, ">= 0.0.0", optional: true} """ {:error, message} end end {:error, _} = error -> error end end ## Implementation below alias RustlerPrecompiled.Config require Logger @checksum_algo :sha256 @checksum_algorithms [@checksum_algo] @native_dir "priv/native" @doc deprecated: "Use available_nifs/1 instead" def available_nif_urls(nif_module) when is_atom(nif_module) do available_nifs(nif_module) |> Enum.map(fn {_lib_name, {url, _headers}} -> url end) end @doc """ Returns URLs for NIFs based on its module name as a list of tuples: `[{lib_name, {url, headers}}]`. The module name is the one that defined the NIF and this information is stored in a metadata file. """ def available_nifs(nif_module) when is_atom(nif_module) do nif_module |> metadata_file() |> read_map_from_file() |> nifs_from_metadata() |> case do {:ok, nifs_with_urls} -> nifs_with_urls {:error, wrong_meta} -> raise "metadata about current target for the module #{inspect(nif_module)} is not available. " <> "Please compile the project again with: `mix compile --force` " <> "Metadata found: #{inspect(wrong_meta, limit: :infinity, pretty: true)}" end end @doc false def nif_urls_from_metadata(metadata) when is_map(metadata) do with {:ok, nifs} <- nifs_from_metadata(metadata) do {:ok, Enum.map(nifs, fn {_lib_name, {url, _headers}} -> url end)} end end @doc false def nifs_from_metadata(metadata) when is_map(metadata) do case metadata do %{ targets: targets, base_url: base_url, basename: basename, nif_versions: nif_versions, version: version } -> all_tar_gzs = for target_triple <- targets, nif_version <- nif_versions do tar_gz_urls( base_url, basename, version, nif_version, target_triple, metadata[:variants] ) end {:ok, List.flatten(all_tar_gzs)} wrong_meta -> {:error, wrong_meta} end end defp maybe_variants_tar_gz_urls(nil, _, _, _), do: [] defp maybe_variants_tar_gz_urls(variants, base_url, target_triple, lib_name) when is_map_key(variants, target_triple) do variants = Map.fetch!(variants, target_triple) for variant <- variants do lib_name = lib_name_with_ext(target_triple, lib_name <> "--" <> Atom.to_string(variant)) {lib_name, tar_gz_file_url(base_url, lib_name)} end end defp maybe_variants_tar_gz_urls(_, _, _, _), do: [] @doc deprecated: "Use current_target_nifs/1 instead" def current_target_nif_urls(nif_module) when is_atom(nif_module) do nif_module |> current_target_nifs() |> Enum.map(fn {_lib_name, {url, _headers}} -> url end) end @doc """ Returns the file URLs to be downloaded for current target as a list of tuples: `[{lib_name, {url, headers}}]`. It is in the plural because a target may have some variants for it. It receives the NIF module. """ def current_target_nifs(nif_module) when is_atom(nif_module) do metadata = nif_module |> metadata_file() |> read_map_from_file() case metadata do %{base_url: base_url, target: target} -> [nif_version, target_triple] = parts_from_nif_target(target) tar_gz_urls( base_url, metadata[:basename], metadata[:version], nif_version, target_triple, metadata[:variants] ) _ -> raise "metadata about current target for the module #{inspect(nif_module)} is not available. " <> "Please compile the project again with: `mix compile --force`" end end defp tar_gz_urls(base_url, basename, version, nif_version, target_triple, variants) do lib_name = lib_name(basename, version, nif_version, target_triple) lib_name_with_ext = lib_name_with_ext(target_triple, lib_name) [ {lib_name_with_ext, tar_gz_file_url(base_url, lib_name_with_ext(target_triple, lib_name))} | maybe_variants_tar_gz_urls(variants, base_url, target_triple, lib_name) ] end @doc """ Returns the target triple for download or compile and load. This function is translating and adding more info to the system architecture returned by Elixir/Erlang to one used by Rust. The returned string has the following format: "nif-NIF_VERSION-ARCHITECTURE-VENDOR-OS-ABI" ## Examples iex> RustlerPrecompiled.target() {:ok, "nif-2.16-x86_64-unknown-linux-gnu"} iex> RustlerPrecompiled.target() {:ok, "nif-2.15-aarch64-apple-darwin"} """ def target( config \\ target_config(), available_targets \\ Config.default_targets(), available_nif_versions \\ Config.available_nif_versions() ) do arch_os = case config.os_type do {:unix, _} -> config.target_system |> normalize_arch_os() |> system_arch_to_string() {:win32, _} -> existing_target = config.target_system |> system_arch_to_string() # For when someone is setting "TARGET_*" vars on Windows if existing_target in available_targets do existing_target else # 32 or 64 bits arch = case config.word_size do 4 -> "i686" 8 -> "x86_64" _ -> "unknown" end config.target_system |> Map.put_new(:arch, arch) |> Map.put_new(:vendor, "pc") |> Map.put_new(:os, "windows") |> Map.put_new(:abi, "msvc") |> system_arch_to_string() end end cond do arch_os not in available_targets -> {:error, "precompiled NIF is not available for this target: #{inspect(arch_os)}.\n" <> "The available targets are:\n - #{Enum.join(available_targets, "\n - ")}"} config.nif_version not in available_nif_versions -> {:error, "precompiled NIF is not available for this NIF version: #{inspect(config.nif_version)}.\n" <> "The available NIF versions are:\n - #{Enum.join(available_nif_versions, "\n - ")}"} true -> {:ok, "nif-#{config.nif_version}-#{arch_os}"} end end defp target_config(available_nif_versions \\ Config.available_nif_versions()) do current_nif_version = :erlang.system_info(:nif_version) |> List.to_string() nif_version = case find_compatible_nif_version(current_nif_version, available_nif_versions) do {:ok, vsn} -> vsn :error -> # In case of error, use the current so we can tell the user. current_nif_version end current_system_arch = system_arch() %{ os_type: :os.type(), target_system: maybe_override_with_env_vars(current_system_arch), word_size: :erlang.system_info(:wordsize), nif_version: nif_version } end # In case one is using this lib in a newer OTP version, we try to # find the latest compatible NIF version. @doc false def find_compatible_nif_version(vsn, available) do if vsn in available do {:ok, vsn} else [major, minor | _] = parse_version(vsn) available |> Enum.map(&parse_version/1) |> Enum.filter(fn [^major, available_minor | _] when available_minor <= minor -> true [_ | _] -> false end) |> case do [] -> :error match -> {:ok, match |> Enum.max() |> Enum.join(".")} end end end defp parse_version(vsn) do vsn |> String.split(".") |> Enum.map(&String.to_integer/1) end # Returns a map with `:arch`, `:vendor`, `:os` and maybe `:abi`. defp system_arch do base = :erlang.system_info(:system_architecture) |> List.to_string() |> String.split("-") triple_keys = case length(base) do 4 -> [:arch, :vendor, :os, :abi] 3 -> [:arch, :vendor, :os] _ -> # It's too complicated to find out, and we won't support this for now. [] end triple_keys |> Enum.zip(base) |> Enum.into(%{}) end # The idea is to support systems like Nerves. # See: https://hexdocs.pm/nerves/compiling-non-beam-code.html#target-cpu-arch-os-and-abi @doc false def maybe_override_with_env_vars(original_sys_arch, get_env \\ &System.get_env/1) do envs_with_keys = [ arch: "TARGET_ARCH", vendor: "TARGET_VENDOR", os: "TARGET_OS", abi: "TARGET_ABI" ] updated_system_arch = Enum.reduce(envs_with_keys, original_sys_arch, fn {key, env_key}, acc -> if env_value = get_env.(env_key) do Map.put(acc, key, env_value) else acc end end) # Only replace vendor if remains the same but some other env changed the config. if original_sys_arch != updated_system_arch and original_sys_arch.vendor == updated_system_arch.vendor and updated_system_arch.os == "linux" do Map.put(updated_system_arch, :vendor, "unknown") else updated_system_arch end end defp normalize_arch_os(target_system) do cond do target_system.os =~ "darwin" -> arch = with "arm" <- target_system.arch, do: "aarch64" %{target_system | arch: arch, os: "darwin"} target_system.os =~ "linux" -> arch = normalize_arch(target_system.arch) vendor = with vendor when vendor in ~w(pc redhat suse alpine) <- target_system.vendor, do: "unknown" %{target_system | arch: arch, vendor: vendor} target_system.os =~ "freebsd" -> arch = normalize_arch(target_system.arch) vendor = with "portbld" <- target_system.vendor, do: "unknown" %{target_system | arch: arch, vendor: vendor, os: "freebsd"} true -> target_system end end defp normalize_arch("amd64"), do: "x86_64" defp normalize_arch("riscv64"), do: "riscv64gc" defp normalize_arch(arch), do: arch defp system_arch_to_string(system_arch) do values = for key <- [:arch, :vendor, :os, :abi], value = system_arch[key], do: value Enum.join(values, "-") end # Calculates metadata based in the TARGET and options # from `config`. # In case target cannot be resolved and "force build" is enabled, # returns only the basic metadata. @doc false def build_metadata(%Config{} = config) do basic_metadata = %{ base_url: config.base_url, crate: config.crate, otp_app: config.otp_app, targets: config.targets, variants: variants_for_metadata(config.variants), nif_versions: config.nif_versions, version: config.version } case target(target_config(config.nif_versions), config.targets, config.nif_versions) do {:ok, target} -> basename = config.crate || config.otp_app [nif_version, target_triple] = parts_from_nif_target(target) lib_name = "#{lib_name(basename, config.version, nif_version, target_triple)}#{variant_suffix(target_triple, config)}" file_name = lib_name_with_ext(target, lib_name) # `cache_base_dir` is a "private" option used only in tests. cache_dir = cache_dir(config.base_cache_dir, "precompiled_nifs") cached_tar_gz = Path.join(cache_dir, file_name) {:ok, Map.merge(basic_metadata, %{ cached_tar_gz: cached_tar_gz, basename: basename, lib_name: lib_name, file_name: file_name, target: target })} {:error, _} = error -> if config.force_build? do {:ok, basic_metadata} else error end end end defp variants_for_metadata(variants) do Map.new(variants, fn {target, values} -> {target, Keyword.keys(values)} end) end # Extract the target without the nif-NIF-VERSION part defp parts_from_nif_target(nif_target) do ["nif", nif_version, triple] = String.split(nif_target, "-", parts: 3) [nif_version, triple] end defp variant_suffix(target, %{variants: variants} = config) when is_map_key(variants, target) do variants = Map.fetch!(variants, target) callback = fn {_name, func} -> if is_function(func, 1) do func.(config) else func.() end end case Enum.find(variants, callback) do {name, _} -> "--" <> Atom.to_string(name) nil -> "" end end defp variant_suffix(_, _), do: "" # Perform the download or load of the precompiled NIF # It will look in the "priv/native/otp_app" first, and if # that file doesn't exist, it will try to fetch from cache. # In case there is no valid cached file, then it will try # to download the NIF from the provided base URL. # # The `metadata` is a map built by `build_metadata/1` and # has details about what is the current target and where # to save the downloaded tar.gz. @doc false def download_or_reuse_nif_file(%Config{} = config, metadata) when is_map(metadata) do name = config.otp_app native_dir = Application.app_dir(name, @native_dir) lib_name = Map.fetch!(metadata, :lib_name) cached_tar_gz = Map.fetch!(metadata, :cached_tar_gz) cache_dir = Path.dirname(cached_tar_gz) file_name = Map.fetch!(metadata, :file_name) lib_file = Path.join(native_dir, file_name) base_url = config.base_url nif_module = config.module result = %{ load?: true, load_from: {name, Path.join("priv/native", lib_name)}, load_data: config.load_data } if File.exists?(cached_tar_gz) do # Remove existing NIF file so we don't have processes using it. # See: https://github.com/rusterlium/rustler/blob/46494d261cbedd3c798f584459e42ab7ee6ea1f4/rustler_mix/lib/rustler/compiler.ex#L134 File.rm(lib_file) with :ok <- check_file_integrity(cached_tar_gz, nif_module), :ok <- :erl_tar.extract(cached_tar_gz, [:compressed, cwd: Path.dirname(lib_file)]) do Logger.debug("Copying NIF from cache and extracting to #{lib_file}") {:ok, result} end else dirname = Path.dirname(lib_file) tar_gz_url = tar_gz_file_url(base_url, lib_name_with_ext(cached_tar_gz, lib_name)) with :ok <- File.mkdir_p(cache_dir), :ok <- File.mkdir_p(dirname), {:ok, tar_gz} <- with_retry(fn -> download_nif_artifact(tar_gz_url) end, config.max_retries), :ok <- File.write(cached_tar_gz, tar_gz), :ok <- check_file_integrity(cached_tar_gz, nif_module), :ok <- :erl_tar.extract({:binary, tar_gz}, [:compressed, cwd: Path.dirname(lib_file)]) do Logger.debug("NIF cached at #{cached_tar_gz} and extracted to #{lib_file}") {:ok, result} end end end defp checksum_map(nif_module) when is_atom(nif_module) do nif_module |> checksum_file() |> read_map_from_file() end defp check_file_integrity(file_path, nif_module) when is_atom(nif_module) do nif_module |> checksum_map() |> check_integrity_from_map(file_path, nif_module) end # It receives the map of %{ "filename" => "algo:checksum" } with the file path @doc false def check_integrity_from_map(checksum_map, file_path, nif_module) do with {:ok, {algo, hash}} <- find_checksum(checksum_map, file_path, nif_module), :ok <- validate_checksum_algo(algo), do: compare_checksum(file_path, algo, hash) end defp find_checksum(checksum_map, file_path, nif_module) do basename = Path.basename(file_path) case Map.fetch(checksum_map, basename) do {:ok, algo_with_hash} -> [algo, hash] = String.split(algo_with_hash, ":") algo = String.to_existing_atom(algo) {:ok, {algo, hash}} :error -> {:error, "the precompiled NIF file does not exist in the checksum file. " <> "Please consider run: `mix rustler_precompiled.download #{inspect(nif_module)} --only-local` to generate the checksum file."} end end defp validate_checksum_algo(algo) do if algo in @checksum_algorithms do :ok else {:error, "checksum algorithm is not supported: #{inspect(algo)}. " <> "The supported ones are:\n - #{Enum.join(@checksum_algorithms, "\n - ")}"} end end defp compare_checksum(file_path, algo, expected_checksum) do case File.read(file_path) do {:ok, content} -> file_hash = algo |> :crypto.hash(content) |> Base.encode16(case: :lower) if file_hash == expected_checksum do :ok else {:error, "the integrity check failed because the checksum of files does not match"} end {:error, reason} -> {:error, "cannot read the file for checksum comparison: #{inspect(file_path)}. " <> "Reason: #{inspect(reason)}"} end end defp cache_dir(sub_dir) do global_cache_path = System.get_env("RUSTLER_PRECOMPILED_GLOBAL_CACHE_PATH") if global_cache_path do Logger.info( "Using global cache for rustler precompiled artifacts. Path: #{global_cache_path}" ) global_cache_path else cache_opts = if System.get_env("MIX_XDG"), do: %{os: :linux}, else: %{} :filename.basedir(:user_cache, Path.join("rustler_precompiled", sub_dir), cache_opts) end end # This arity is only used in test context. It should be private because # we can't provide this option in the `mix rustler_precompiled.download` task. defp cache_dir(basedir, sub_dir) do if basedir do Path.join(basedir, sub_dir) else cache_dir(sub_dir) end end defp lib_prefix(target) do if String.contains?(target, "windows") do "" else "lib" end end defp lib_name(basename, version, nif_version, target_triple) do "#{lib_prefix(target_triple)}#{basename}-v#{version}-nif-#{nif_version}-#{target_triple}" end defp lib_name_with_ext(target, lib_name) do ext = if String.contains?(target, "windows") do "dll" else "so" end "#{lib_name}.#{ext}.tar.gz" end defp tar_gz_file_url({module, function_name}, file_name) when is_atom(module) and is_atom(function_name) do apply(module, function_name, [file_name]) end defp tar_gz_file_url({base_url, request_headers}, file_name) do uri = URI.parse(base_url) uri = Map.update!(uri, :path, fn path -> Path.join(path || "", file_name) end) {to_string(uri), request_headers} end defp tar_gz_file_url(base_url, file_name) do tar_gz_file_url({base_url, []}, file_name) end defp download_nif_artifact(url) when is_binary(url) do download_nif_artifact({url, []}) end defp download_nif_artifact({url, request_headers}) do url = String.to_charlist(url) Logger.debug("Downloading NIF from #{url}") {:ok, _} = Application.ensure_all_started(:inets) {:ok, _} = Application.ensure_all_started(:ssl) proxy = System.get_env("HTTP_PROXY") || System.get_env("http_proxy") with true <- is_binary(proxy), %{host: host, port: port} when is_binary(host) and is_integer(port) <- URI.parse(proxy) do Logger.debug("Using HTTP_PROXY: #{proxy}") :httpc.set_options([{:proxy, {{String.to_charlist(host), port}, []}}]) end proxy = System.get_env("HTTPS_PROXY") || System.get_env("https_proxy") with true <- is_binary(proxy), %{host: host, port: port} when is_binary(host) and is_integer(port) <- URI.parse(proxy) do Logger.debug("Using HTTPS_PROXY: #{proxy}") :httpc.set_options([{:https_proxy, {{String.to_charlist(host), port}, []}}]) end # https://erlef.github.io/security-wg/secure_coding_and_deployment_hardening/inets # respects the user provided ca certs via Hex env var cacertfile = System.get_env("HEX_CACERTS_PATH", CAStore.file_path()) http_options = [ ssl: [ verify: :verify_peer, cacertfile: cacertfile |> String.to_charlist(), # We need to increase depth because the default value is 1. # See: https://erlef.github.io/security-wg/secure_coding_and_deployment_hardening/ssl depth: 3, customize_hostname_check: [ match_fun: :public_key.pkix_verify_hostname_match_fun(:https) ] ] ] options = [body_format: :binary] request_headers = Enum.map(request_headers, fn {k, v} when is_binary(k) -> {String.to_charlist(k), v} end) case :httpc.request(:get, {url, request_headers}, http_options, options) do {:ok, {{_, 200, _}, _headers, body}} -> {:ok, body} other -> {:error, "couldn't fetch NIF from #{url}: #{inspect(other)}"} end end # Download a list of files from URLs and calculate its checksum. # Returns a list with details of the download and the checksum of each file. @doc false def download_nif_artifacts_with_checksums!(nifs_with_urls, options \\ []) do ignore_unavailable? = Keyword.get(options, :ignore_unavailable, false) attempts = max_retries(options) download_results = for {lib_name, url} <- nifs_with_urls, do: {lib_name, with_retry(fn -> download_nif_artifact(url) end, attempts)} cache_dir = cache_dir("precompiled_nifs") :ok = File.mkdir_p(cache_dir) Enum.flat_map(download_results, fn result -> with {:download, {lib_name, download_result}} <- {:download, result}, {:download_result, {:ok, body}} <- {:download_result, download_result}, hash <- :crypto.hash(@checksum_algo, body), path <- Path.join(cache_dir, lib_name), {:file, :ok} <- {:file, File.write(path, body)} do checksum = Base.encode16(hash, case: :lower) Logger.debug( "NIF cached at #{path} with checksum #{inspect(checksum)} (#{@checksum_algo})" ) [ %{ lib_name: lib_name, path: path, checksum: checksum, checksum_algo: @checksum_algo } ] else {:file, error} -> raise "could not write downloaded file to disk. Reason: #{inspect(error)}" {context, result} -> if ignore_unavailable? do Logger.debug( "Skip an unavailable NIF artifact. " <> "Context: #{inspect(context)}. Reason: #{inspect(result)}" ) [] else raise "could not finish the download of NIF artifacts. " <> "Context: #{inspect(context)}. Reason: #{inspect(result)}" end end end) end defp max_retries(options) do value = Keyword.get(options, :max_retries, 3) if value not in 0..15, do: raise("attempts should be between 0 and 15. Got: #{inspect(value)}") value end defp with_retry(fun, attempts) when attempts in 0..15 do first_try = fun.() Enum.reduce_while(1..attempts//1, first_try, fn count, partial_result -> case partial_result do {:ok, _} -> {:halt, partial_result} err -> Logger.info("Attempt #{count} failed with #{inspect(err)}") wait_in_ms = :rand.uniform(count * 2_000) Process.sleep(wait_in_ms) {:cont, fun.()} end end) end defp read_map_from_file(file) do with {:ok, contents} <- File.read(file), {%{} = contents, _} <- Code.eval_string(contents) do contents else _ -> %{} end end defp write_metadata(nif_module, metadata) do metadata_file = metadata_file(nif_module) existing = read_map_from_file(metadata_file) if Map.equal?(metadata, existing) do :ok else dir = Path.dirname(metadata_file) :ok = File.mkdir_p(dir) File.write(metadata_file, inspect(metadata, limit: :infinity, pretty: true)) end end defp metadata_file(nif_module) when is_atom(nif_module) do rustler_precompiled_cache = cache_dir("metadata") Path.join(rustler_precompiled_cache, "metadata-#{nif_module}.exs") end # Write the checksum file with all NIFs available. # It receives the module name and checksums. @doc false def write_checksum!(nif_module, checksums) when is_atom(nif_module) do metadata = nif_module |> metadata_file() |> read_map_from_file() case metadata do %{otp_app: _name} -> file = checksum_file(nif_module) pairs = for %{path: path, checksum: checksum, checksum_algo: algo} <- checksums, into: %{} do basename = Path.basename(path) checksum = "#{algo}:#{checksum}" {basename, checksum} end lines = for {filename, checksum} <- Enum.sort(pairs) do ~s( "#{filename}" => #{inspect(checksum, limit: :infinity)},\n) end File.write!(file, ["%{\n", lines, "}\n"]) _ -> raise "could not find the OTP app for #{inspect(nif_module)} in the metadata file. " <> "Please compile the project again with: `mix compile --force`." end end defp checksum_file(nif_module) do # Saves the file in the project root. Path.join(File.cwd!(), "checksum-#{nif_module}.exs") end end