The Merits of Independent Audit of AI Systems

For those who have followed ForHumanity, you know that we have been developing and promoting the concept of Independent Audit of AI systems. If you are new to the concept of Independent Audit, the basics can be found here:

AI Safety — The Concept of Independent Audit
For months I have regularly tweeted a response to my colleagues in the AI Safety community about being a supporter of…

and here:

SAFEAI — A Tool for Concerned Parents
What is SAFEAI?

For this post, I wanted to cover the benefits of Indepedent Audit which are maximized when four things occur:

  1. Corporations widely adopt Independent Audit

  2. Governments make Independent Audit mandatory, akin to the requirements related to GAAP or IFRS accounting principles

  3. AI Safety professionals widely participates in the open-source, crowd-sourced search for best-practices

  4. When consumers of product/services use the SAFEAI logo to inform their buying/usage decisions

It is under these assumptions that I will talk about the features and subsequent benefits which accrue to humanity. Below is a bullet point list:

  1. Transparency — consumers of both products and services will receive an unprecendent level of transparency from companies. The transparency/disclosure we refer to here is in a few key areas, such as ethical decision-making, data usage, safety, control, explainability of algorithms, accountability of both algorithms and corporations, and bias avoidance. These areas of increased Transparency result in…

  2. Fairer markets — when markets are transparent then decisions made in the marketplace will be better. Rewarding responsible companies and punishing irresponsible companies. To be clear, it is the market that rewards and punishes, based on transparency and choice. And…

  3. Trust — this is the foundation of fairer markets. A marketplace which has become increasingly adversarial between companies and consumers, especially in the capture and exploitation of data, can be reversed. Trust is engendered when consumers feel that they are being provided with a valuable service AND when the price for that service is considered fair. Price, in this case, includes not only monetary compensation for services, but also impacts to privacy and personal well-being.

  4. Opacity — This benefit accrues to corporations, but has an over-arching, related benefit, to society. Opacity is the opposite of Transparency (referenced above), so we must explain the why both are listed as features. Transparency is discussed above, but does not extend to the intellectual property of the company — specifically, the code and machines that are employing artifical intelligence. Opacity to corporations allows them to protect their intellectual property (IP). When IP can be protected, then companies will invest in products and service development, knowing that they can recoop their costs and earn a fair profit. When transparency is EXCESSIVE (often due to regulation), then investment is discouraged and in fact, cheating, copying and outright theft become commonplace. Choosing Independent Audit allows a company to protect its IP without excessive disclosure. Choosing Independent Audit, might offset the increasing call for comprehensive disclosure that legal authorities might mandate. Essentially, Independent Audit strikes the best balance between opacity and transparency, giving corporations the protection they need to justify investment while giving consumers the transparency they need to trust the companies which provide them products/services.

  5. -Third-party verification — The value of third-party verification is an under-valued element in our world today. There are many places, where third-party verification has dramatically raised the bar of quality. Starting with financial audits and ranging to product-testers, such as Consumer Reports. These behind-the-scenes services act as a watch dog on your behalf. There are costs associated with all independent third-party reviews and those costs are passed thru to you, the consumer. The result is that companies know that they cannot cheat, they cannot cut corners, they cannot act unethically or they will be called out. These systems are not perfect for various reasons, not the least of which is a belief by humans that they can “get away with it”, but in the end, the truth will always come out sooner-or later as long as there is a watchdog in the room.

  6. Extends beyond national boundaries — Independent Audit is a market based mechanism. The SAFEAI seal of approval will build brand recognition. Over time, it will effect consumer decision-making, causing consumers to choose products and services which have been auditted for SAFEAI. The impacts of this “seal-of-approval” are not subject to national boundries the way that laws are hindered in their effectiveness. We are not suggesting that countries should not pass laws and regulations in AI Safety, quite to the contrary, new laws and regulations will have great value. However, it is a corporation’s responsbility to avoid regulations and laws when it is legal to do so and will result in greater profitability for the company. Therefore, it is necessary to have a market-based mechanism, one that impacts profitability directly and globally, such as the Independent Audit of AI Systems and the SAFEAI seal-of-approval. When consumers use the SAFEAI logo to inform their purchase and usage decisions then we know that safe and responsible AI will be profitable while dangerous and irresponsible AI will effefcctively make a company’s products worthless. When this happens, humanity wins, but it requires humanity to participate and to pay attention to the SAFEAI logo.

  7. On-going and Dynamic process — unlike standard financial accounting principles which may go unchanged for years at a time, “best-practices” in ethics, bias, privacy, trust and cybersecurity are likely to change regulalry. ForHumanity will maintain a dynamic, transparent and constant review process with our global open-source, crowd-sourced network to continuingly uncover “best-practices” which will update our audit process quarterly.

  8. Transparent process — The globally, open-source, crowd-sourced process is open to all. Anyone may join the conversation. All will be heard, all input will be considered, all votes count and most importantly, all reasonable dissent will be tracked and addressed. As the audit process is created, the results will be transparent to all. Further comment and critique is encouraged so that we may refine our process to achieve the best possible results. There is no fee or membership required to participate in the input process. There are only rules of proper decorum, so that we conduct our business in a civilized manner that protects the rights and dignity of all involved.

  9. Dedicated professionals- ForHumanity maintains a full-time staff dedicated to each of the core audit silos of ethics, bias, privacy, trust and cybersecurity. As a result, these professionals are constantly sourcing new ideas and new contacts to uncover the “best-practices” for our audit process. They maintain daily office hours and are continually reviewing input from all over the world. Our professionals are able to bridge the gap between academic thought and practical application to deliver the dynamic audit process.

  10. Audit process may be tailored for local jurisdictions — not dissimilar to the way accounting principles matured overtime, ForHumanity expects that the audit process will become tailored. Laws, customs and regulations may dictate that the audit process be adjusted slightly on a country-by-country (or regional) basis. This will allow the process to achieve an over-arching comment on “best-practices” while being locally compliant.

  11. Audit is good for one year — Independent Audit is conducted annually using the current audit process for “best-practices”. As these “best-practices” are likely to change fairly regularly as technology advances, this could create an onerous process of compliance for companies. The most recent audit will be good for one year, regardless of changes in “best-practices”. This standard allows the company time to comply with the ever changing “best-practices” while permitting the audit process to remain dynamic.

  12. Results are binary- When a company submits to a SAFEAI audit, it is a pass/fail endeavor on each of the five audit silos. The process will be structured so that companies are either compliant or non-compliant, so that the consumer may know if a company is using “best-practices” for ethics, bias, privacy, trust and cybersecurity. A company may pass any number of the silos, but is not SAFEAI compliant unless all five silos are passed successfully.

  13. Goal-Aligned with Humanity. ForHumanity exists for one purpose, to mitigate the downside risks associated with AI and Automation in order for Humanity to receive the highest possible benefits from these technological advancements. Our client-base, is you. ForHumanity is a non-profit organization designed to serve those who would buy products or use services which rely upon artificial intelligence. ForHumanity has a few sources of revenue. First, we receive funding from individuals through gracious donations. Second, we receive licensing revenue from corporations who have already completed SAFEAI audits when they choose to license the SAFEAI logo to show the world. The company does NOT pay a fee to ForHumanity for the audit and never will. We will not have the audit results tainted by a “pay-for-play” process which incentivizes successful audits. Only once they have successfully complied with the audit, then may a company pay to license the SAFEAI logo to demonstrate successfully passing the audit. Other members pay membership fees if they want to serve humanity, but fixing problems at companies that want to be SAFEAI compliant. Finally, our core members are those which conduct the audits on behalf of ForHumanity and thus for you. In the end, we strive to ensure that the goals of ForHumanity are prefectly aligned to serve… humanity.

ForHumanity believes that Independent Audit of AI Systems is crucial to mitigating the downside risks associated with the proliferation of AI and automation. But it doesn’t exist in a vacuum and it doesn’t succeed without the help of everyone. Please consider how you can become invovled and reach out.