Bitlend
Search…
⌃K

Emergency Procedure

Introduction

The purpose of this document is to outline the steps Bitlend will take during an emergency situation. Bitlend’s intent is to minimize the risk of loss for its users and treasury.

Defining An Emergency

For the purposes of this document, an emergency situation is defined as any event that could potentially lead to the loss of funds for Bitlend’s users, treasury, or contracts that Bitlend has deployed.
Examples of an emergency are as follows:
  1. 1.
    A bug or code exploit that leads to loss of funds.
  2. 2.
    The loss of a private key for a primary role (e.g. multisig holder).
  3. 3.
    An exploit discovered by a core team member or bug bounty participant.
  4. 4.
    An active exploit or hack discovered by a third party.

Roles

In the event of an emergency, members of the Bitlend team will be assigned the following roles:
  • Facilitator
  • Multi-Sig herder
  • Core Dev Lead
  • Web Lead
  • Ops
  • BTTC DAO Guardian

Facilitator

A facilitator is a member who ensures that the procedures outlined in this document are followed closely and in a timely manner. They coordinate between relevant parties to assist in quick decision-making. A person is qualified to be a facilitator if they are familiar with the emergency procedures, and if they have experience in a similar role.

Multi-Sig Herder

The Multisig herder is responsible for ensuring that the Bitlend team’s multi-sig wallets are ready to execute transactions in a timely manner during the emergency. They are tasked with coordinating communications between multisig signers, preparing and helping to execute transactions.

Core Dev Lead

The Core Dev Lead is responsible for quick changes to governance and guardian roles during the emergency. The Core Dev Lead also coordinates quick changes to the web UI as required. Examples of such responsibilities are:
  • Preparing and executing core dev multi-sig operations and transactions
  • Setting vaults into emergency shutdown mode
  • Enabling / disabling deposits and withdrawals if necessary
  • Creating alert displays
Ops
Responsible for creating communications and assisting operations as needed.
  • Coordinates communications
  • Takes notes of key events and timelines for disclosure
BTTC DAO Guardian
The role of the BTTC DAO Guardian is to act as a representative of the BTTC DAO and to steward the BitLend protocol along with its governance. The BTTC DAO Guardian will have the same permissions as proposal and timelock. This includes the ability to create proposals, cancel proposals, modify the Price Oracle, add new guardians, modify the BorrowCap of all markets, set parameters of the interest rate model, and pause the market/protocol.
The BTTC DAO Guardian will play a crucial role in maintaining the stability and security of the BitLend platform by overseeing and managing the protocol's operations. They will have the authority to make important decisions and take actions in response to potential risks and emergencies, in order to safeguard the interests of the users and the treasury.
The BTTC DAO Guardian will work closely with the BitLend team and community to ensure that the platform is operated in a transparent and fair manner. They will have access to all relevant information and tools to make informed decisions and take appropriate actions in the best interests of the community.

Emergency Steps

This section of the document will act as a guideline to follow if a reported incident requires immediate attention.
All decisions should be driven toward the goal of minimizing loss of funds, under the guidance of the BTTC DAO.
  1. 1.
    Open a private chat room (war room) with a voice channel. Online team members who are capable of fulfilling the roles outlined above should be invited. Only members who are acting in the specified roles should be included in the war room, as well as others who can provide insight into the circumstances of the emergency.
  2. 2.
    All information in the war room is considered confidential and should not be shared with third parties. Important data should be pinned and updated by the facilitator.
  3. 3.
    The first objective of the team is to assess the situation. This should be done in a timely manner - confirming relevant information and assessing the situation’s severity.
    1. 1.
      Is the issue valid? Can several team members confirm? Is there evidence of transactions that confirm the incident? Such evidence should be pinned.
    2. 2.
      Is there a code expert in the war room?
    3. 3.
      Are funds presently at risk? What immediate actions, if any, are required?
    4. 4.
      Is the issue isolated to one market, or are multiple markets affected? Pin all affected contracts.
    5. 5.
      What multi-sigs are required to address the issue? The multi-sig herder should notify signers and clear the queue in preparation for emergency transactions.
    6. 6.
      If there is no immediate risk to funds, what actions are necessary to mitigate and/or take preventative action?
    7. 7.
      Is the team in agreement that the situation is under control and the war room should be closed?
    8. 8.
      Once the situation has been assessed as valid, immediate corrective action must be taken to prevent the additional loss of funds. In this situation, it is best to err on the side of caution. Should deposits be disabled? Are there multiple team members able to confirm corrective actions will mitigate risk? What other actions, if any, should be taken?
  4. 4.
    Once the situation has been assessed as valid, immediate corrective action must be taken to prevent the additional loss of funds under the guidance of the BTTC DAO.
    The BTTC DAO will be informed of the incident and all actions taken by the team during the emergency.
a) Should deposits be disabled?
b) Are there multiple team members able to confirm corrective actions will mitigate r risk?
c) What other actions, if any, should be taken?
5. Once corrective measures are in place and funds are no longer at risk, the root cause of the incident must be identified. Additional things to consider at this phase are:
a) What communications should be made public at this point?
b) Can research be divided among war room members, sharing debug screens to quickly identify root cause?
6. Once the root issue has been identified, the team should formulate a redemption plan and its code implementation. How can solutions be weighed to mitigate further loss of funds? Can solutions be tested to confirm the root issue is resolved? Does the war room agree on a solution? If not, identify all objections and work toward consensus - prioritizing minimization of losses. Does the solution require a long-term plan? Once a solution has been implemented, the team is to confirm that the resolution is successful and funds are safe. Possible actions: Run code simulations to confirm the efficacy of the proposed solution Coordinate signatures from multi-sig signers and execute the solution Enable any necessary UI changes Assign a member of the war room to prepare a disclosure and a timeline of events that took place. The team agrees that the war room can be closed.

Emergency Checklist

  • Create war room with audio
  • Assign key roles to war room members
  • Add code expert to the war room
  • Clear multi-sig queues
  • Disable deposits and/or withdrawals as needed on the UI
  • Confirm and identify root issue
  • Take immediate preventative action to prevent further loss
  • Communicate the situation internally and externally
  • Propose workable solutions
  • Prioritize solutions
  • Implement solutions
  • Reach consensus within the team on a best solution
  • Implement solution
  • Confirm incident is resolved
  • Assign someone to produce a disclosure report
  • Disband war room
  • Schedule post mortem

Post Mortem

A post mortem should be conducted post-incident and should gather all data and feedback from war-room participants. This data should be used to improve the processes laid out in this document.
The facilitator should conduct an informal debrief and gather initial notes before they are forgotten by participants.
These notes should be compiled into an extensive post-mortem report with the following inputs:
  • List of what went well
  • List of what could be improved
  • List of questions that came up in the post-mortem
  • List of insights from the process
  • Root cause analysis to prevent a recurrence of the incident
  • List of action items and owners with estimates for completion
The facilitator should track completion of these action items. Examples of action items are:
  • Changes in process and documentation
  • Changes in code and testing
  • Changes in tools, etc.

Conclusion

The BitLend and BTTC DAO are committed to ensuring the safety and security of our users' funds and the stability of our platform. The emergency procedures outlined in this document are in place to minimize the risk of loss for our users and treasury during an emergency situation. The BTTC DAO's oversight, support and additional safety-layer for the community allows for consistent and reliable decision making during these critical times. It is important for all users to be familiar with these procedures and to understand the potential risks associated with web3 lending platforms. As always, it is the responsibility of users to carefully evaluate and manage their own risk.
​