im server
1.0.0
Instant messaging applications, including server, management and client
It has been deployed and launched, welcome to experience the client and management side
Please do not change the default role and permissions at will. Please feel a little loving and don't make up some uncivilized names.
Using the egg framework, the server side of IM service
Since the development of mobile Internet, instant messaging services led by WeChat have been integrated into every corner of our lives and also play an important role in some of the company's businesses. Our company used instant messaging services, but many customized needs cannot be achieved. Therefore, it was decided to develop an instant messaging microservice that meets customized needs internally.
The socket.io framework was used because the backend was short of people at that time. After reading some examples, I thought it was really convenient to use and supported on the entire platform. Therefore, this microservice was implemented in the front-end team, and the effect is currently pretty good.
The community currently has less or too simple content in this area (there is only one public chat room). In addition, it is very uncomfortable to be PM during the business development process, so I want to break away from some unique business things and realize a simple and complete IM application that does not mix company business with simple functions, including server, management and client. The object of imitation of the client is WeChat, because I am very familiar with it and don’t have to think too much about the product. In addition, the people who try it out are very familiar with it and do not require too much communication costs.
To develop a complete set of instant messaging services, the following parts are required:
Born for enterprise-level frameworks and applications
The reason I chose Alibaba's egg.js framework as support is that they have done a good job in large-scale implementation and security within it. The reason why they did not choose nest is that it is troublesome to integrate socket.io . ORM uses sequelize, and the database is mysql. I have used it before, so it is less difficult to get started.
Out-of-the-box front-end/design solution
The reason for choosing Ant Design Pro is that I am familiar with Vue Family Bucket. I want to take this opportunity to familiarize myself with the development process of the entire React ecosystem and feel the essential differences and different paths of the two major development frameworks in China. Ant Design Pro has been released for several years, and it has indeed brought efficiency improvements to small and medium-sized enterprises, which is just right for my needs.
Standard tools developed by Vue.js
Use @vue/cli to build an IM service client, a mobile H5 project, and the UI framework uses Youzan vant, which integrates my open source component vue-page-stack and Mr. Huang's better-scroll to realize the basic functions of IM
As a front-end engineer, most daily tasks do not require thinking about entity relationships. However, based on my actual experience, understanding entity relationships can help us better understand business models. The improvement of product and business understanding is of great help to us. We can find many illogical aspects during demand review (why do we have to complain about the product manager again). If we can propose it at this time, we will take the initiative to avoid repeated development in the subsequent process, and at the same time, we can form a relatively good interaction with the students on the product side (rather than confrontation). Here are some of the more important entity relationships:
From the above figure, we can see that user is the core of the entire relationship diagram. The following is the relationship between various entities:
The following is a detailed introduction to sessions, roles and permissions:
When completing an instant messaging application, the first thing you need to consider is the conversation, which is the conversation window in our WeChat. It takes a lot of effort to think about the relationship between conversations and messages, users, and groups, and finally form the following basic relationship:
In other words, the user has no direct relationship with the session, and can only obtain the session through the user's corresponding single chat and group chat. This can have the following benefits:
In order to design a flexible, universal and convenient permission management system, this system adopts RBAC (role-based access control) control to design a general "user role permissions" platform for convenience of later expansion.
RBAC (role-based access control) refers to the user being associated with permissions through roles. That is, a user has several roles, and each role has several permissions (of course, don't combine conflicting roles and permissions together). In this way, an authorization model of "user-role-permissions" is constructed. In this model, there is generally a many-to-many relationship between users and roles and between roles and permissions.
This system has the default roles of administrator, general user, banned user and banned user, and different permissions are assigned to different roles. Therefore, it is necessary to perform unified authentication (through middleware) processing for interface routing such as management and speech. The specific methods and methods will be explained in detail in the back-end project. This system temporarily uses the method of predefined roles and permissions. If you want to expand in the future, you can edit roles and permissions.
I have never seen the management side of WeChat, but you can imagine that administrators can configure user roles and permissions and edit group status:
After registering and logging in, you can add friends and join groups normally, modify personal basic information and process applications.
Unable to log in
For example: Now there is a new version of the Personal Center that needs to be tested online. First, create a new role "Test Personal Center", and then assign corresponding permissions to the role; then group ordinary users and select some people to configure this role, so that you can test it.
Let’s talk about the core communication principle of instant messaging services. Like general http services, there is a server and client for communication, but the detailed protocol and processing methods are different.
Due to historical reasons, the mainstream http protocol is now a stateless protocol (HTTP2 is not widely used for the time being). Generally, the client initiates the request actively and then responds to it. So in order to realize that the server pushes information to the client, the front-end needs to actively poll the back-end. This method is inefficient and error-prone. This is indeed done on our management homepage before (5s once).
In order to achieve this need for server-side to actively push information, HTML5 began to provide a protocol for full-duplex communication on a single TCP connection, namely WebSocket. WebSocket makes data exchange between clients and servers easier, allowing the server to actively push data to clients. The WebSocket protocol was born in 2008 and became an international standard in 2011. Currently, most browsers have already supported it.
The usage of WebSocket is quite simple:
var ws = new WebSocket("wss://echo.websocket.org");
ws.onopen = function(evt) {
console.log("Connection open ...");
ws.send("Hello WebSockets!");
};
ws.onmessage = function(evt) {
console.log( "Received Message: " + evt.data);
ws.close();
};
ws.onclose = function(evt) {
console.log("Connection closed.");
};
With the WebSocket protocol, the server has advanced weapons to actively push information. So is there any way to be compatible with old and new browsers? In fact, many people think of this, and the answer is socket.io
socket.io socket.io further encapsulates the WebSocket interface, and can automatically switch to the use of polling in old browsers for communication (our users will not perceive), forming a unified set of interfaces, greatly reducing the burden of development. It mainly has the following advantages:
This is the socket.io homepage
The fastest and most reliable instant messaging engine (FEATURING THE FASTEST AND MOST RELIABLE REAL-TIME ENGINE)
It's really easy to use:
var io = require('socket.io')(80);
var cfg = require('./config.json');
var tw = require('node-tweet-stream')(cfg);
tw.track('socket.io');
tw.track('javascript');
tw.on('tweet', function(tweet){
io.emit('tweet', tweet);
});
Let’s focus on several important points in the Server side project. Most of the content needs to be viewed on the egg official website.
Use scaffolding npm init egg --type=simple to initialize the server project, install mysql (my is version 8.0), configure the database link password required for sequelize, etc., and you can start it
// 目录结构说明
├── package.json // 项目信息
├── app.js // 启动文件，其中有一些钩子函数
├── app
| ├── router.js // 路由
│ ├── controller
│ ├── service
│ ├── middleware // 中间件
│ ├── model // 实体模型
│ └── io // socket.io 相关
│ ├── controller
│ └── middleware // io独有的中间件
├── config // 配置文件
| ├── plugin.js // 插件配置文件
| └── config.default.js // 默认的配置文件
├── logs // server运行期间产生的log文件
└── public // 静态文件和上传文件目录
Router is mainly used to describe the correspondence between the request URL and the Controller that specifically undertakes the execution action, that is, app/router
app/middleware/auth.jsapp/middleware/admin.js Because this system has different roles as administrators and general communication users, unified authentication processing is required for the interface routing of management and communication.
For example, the management side route /v1/admin/... , I want to add administrator authentication to all routes in this series. At this time, you can use middleware to authenticate. The following is a specific example of using middleware in admin router.
// middware
module.exports = () => {
return async function admin(ctx, next) {
let { session } = ctx;
// 判断admin权限
if (session.user && session.user.rights.some(right => right.keyName === 'admin')) {
await next();
} else {
ctx.redirect('/login');
}
};
};
// router
const admin = app.middleware.admin();
router.get('/api/v1/admin/rights', admin, controller.v1.admin.rightsIndex);
The sequelize+mysql combination used, and egg also has sequelize related plug-ins. sequelize is an ORM used in Node environment, supporting Postgres, MySQL, MariaDB, SQLite and Microsoft SQL Server, which is quite convenient to use. You need to define the direct relationship between the model and the model first. After the relationship is established, you can use some preset methods.
The basic information of the model is easier to process. What needs to be paid attention to is the relationship design between entities, that is, associate. The following is the relationship description of user
// User.js
module.exports = app => {
const { STRING } = app.Sequelize;
const User = app.model.define('user', {
provider: {
type: STRING
},
username: {
type: STRING,
unique: 'username'
},
password: {
type: STRING
}
});
User.associate = function() {
// One-To-One associations
app.model.User.hasOne(app.model.UserInfo);
// One-To-Many associations
app.model.User.hasMany(app.model.Apply);
// Many-To-Many associations
app.model.User.belongsToMany(app.model.Group, { through: 'user_group' });
app.model.User.belongsToMany(app.model.Role, { through: 'user_role' });
};
return User;
};
For example, the relationship between user and userInfo is a one-to-one relationship. After it is defined, we can use user.setUserInfo(userInfo) when creating a new user. When you want to obtain the basic information of this user, you can also use user.getUserInfo()
The relationship between User and Apply is one-to-many, that is, a user can correspond to multiple applications, and currently only friends and group applications are applied:
When adding an application, you can use user.addApply(apply) , and when obtaining it, you can use it as follows:
const result = await ctx.model.Apply.findAndCountAll({
where: {
userId: ctx.session.user.id,
hasHandled: false
}
});
The relationship between user and group is many-to-many, that is, a user can correspond to multiple groups, and a group can correspond to multiple users. In this way, sequelize will establish an intermediate table user_group to achieve this relationship.
I usually use this:
group.addUser(user); // 建立群组和用户的关系
user.getGroups(); // 获取用户的群组信息
egg provides the egg-socket.io plug-in. You need to open the plug-in in config/plugin.js after installing egg-socket.io. io has its own middleware and controller.
The route of io is different from that of general http requests. Note that the route here cannot be processed with middleware (I did not succeed), so I handled the ban processing in the controller.
// 加入群
io.of('/').route('/v1/im/join', app.io.controller.im.join);
// 发送消息
io.of('/').route('/v1/im/new-message', app.io.controller.im.newMessage);
// 查询消息
io.of('/').route('/v1/im/get-messages', app.io.controller.im.getMessages);
Note: I regard both group and friend relationship as a room (that is, a session), so that I can send messages directly to the romm, and everyone inside can receive them.
There are two default middleware, one is the connection Middleware called when connecting and disconnecting, which is used to verify the login status and process business logic; the other is the packet Middleware called every time a message is sent, which is used to print the log
Since the call permission is preset, it is processed in the controller
// 对用户发言的权限进行判断
if (!ctx.session.user.rights.some(right => right.keyName === 'speak')) {
return;
}
Chats are divided into single chats and group chats. The chat information temporarily includes general text, pictures, videos and positioning messages, which can be expanded into orders or products according to the business.
The structure design of message refers to the design of several third-party services, and has also been adjusted in combination with the situation of this project itself. It can be expanded at will, and the following explanation is made:
const Message = app.model.define('message', {
/**
* 消息类型：
* 0:单聊
* 1:群聊
*/
type: {
type: STRING
},
// 消息体
body: {
type: JSON
},
fromId: { type: INTEGER },
toId: { type: INTEGER }
});
The body stores the message body, which is used to store different message formats using json:
// 文本消息
{
"type": "txt",
"msg":"哈哈哈" //消息内容
}
// 图片消息
{
"type": "img",
"url": "http://nimtest.nos.netease.com/cbc500e8-e19c-4b0f-834b-c32d4dc1075e",
"ext":"jpg",
"w":360, //宽
"h":480, //高
"size": 388245
}
// 视频消息
{
"type": 'video',
"url": "http://nimtest.nos.netease.com/cbc500e8-e19c-4b0f-834b-c32d4dc1075e",
"ext":"mp4",
"w":360, //宽
"h":480, //高
"size": 388245
}
// 地理位置消息
{
"type": "loc",
"title":"中国 浙江省 杭州市 网商路 599号", //地理位置title
"lng":120.1908686708565, // 经度
"lat":30.18704515647036 // 纬度
}
There is only one at present, which is to update the token of baidu. It is quite simple here, just refer to the official documentation.
Intelligent dialogue customization and service platform UNIT
This is quite interesting. You can create a new robot and add corresponding skills at https://ai.baidu.com/ . I am chatting here, and there are smart Q&As, etc.
If you do not want to start, you can delete ctx.service.baidu.getToken();
First of all, you need to configure it in the configuration file. I have limited the file size here and cross-site iOS video file format:
config.multipart = {
mode: 'file',
fileSize: '3mb',
fileExtensions: ['.mov']
};
A unified interface is used to handle file uploads. The core problem is file writing, and files are the file list transmitted from the front end
for (const file of ctx.request.files) {
// 生成文件路径，注意upload文件路径需要存在
const filePath = `./public/upload/${
Date.now() + Math.floor(Math.random() * 100000).toString() + '.' + file.filename.split('.').pop()
}`;
const reader = fs.createReadStream(file.filepath); // 创建可读流
const upStream = fs.createWriteStream(filePath); // 创建可写流
reader.pipe(upStream); // 可读流通过管道写入可写流
data.push({
url: filePath.slice(1)
});
}
I store /public/upload/ in the server directory. This directory requires static file configuration:
config.static = {
prefix: '/public/',
dir: path.join(appInfo.baseDir, 'public')
};
The official egg document of this chapter is to kill you, there are nothing examples, you must read the source code. It's so terrible. I have studied it for a long time before I figured out what's going on.
Because I want to control account password login more freely, I do not use passport for account password login, but use ordinary interface authentication and session.
The following is a detailed description of the process of logging in using a third-party platform (I chose GitHub):
Open the plug-in:
// config/plugin.js
module.exports.passport = {
enable: true,
package: 'egg-passport',
};
module.exports.passportGithub = {
enable: true,
package: 'egg-passport-github',
};
// config.default.js
config.passportGithub = {
key: 'your_clientID',
secret: 'your_clientSecret',
callbackURL: 'http://localhost:3000/api/v1/passport/github/callback' // 注意这里非常的关键，这里需要和你在github上面设置的Authorization callback URL一致
};
this.app.passport.verify(verify);
const github = app.passport.authenticate('github', { successRedirect: '/' }); // successRedirect就是最后校验完毕后前端会跳转的路由，我这里直接跳转到主页了
router.get('/v1/passport/github', github);
router.get('/v1/passport/github/callback', github);
/v1/passport/github on the front end and initiate github authorization for this application. After success, github will go to http://localhost:3000/v1/passport/github/callback?code=12313123123 . Our githubPassport plug-in will obtain the user's information on github. After obtaining the detailed information, we need to verify the user information in app/passport/verify.js , and associate it with the user information of our own platform, and also assign a value to the session // verify.js
module.exports = async (ctx, githubUser) => {
const { service } = ctx;
const { provider, name, photo, displayName } = githubUser;
ctx.logger.info('githubUser', { provider, name, photo, displayName });
let user = await ctx.model.User.findOne({
where: {
username: name
}
});
if (!user) {
user = await ctx.model.User.create({
provider,
username: name
});
const userInfo = await ctx.model.UserInfo.create({
nickname: displayName,
photo
});
const role = await ctx.model.Role.findOne({
where: {
keyName: 'user'
}
});
user.setUserInfo(userInfo);
user.addRole(role);
await user.save();
}
const { rights, roles } = await service.user.getUserAttribute(user.id);
// 权限判断
if (!rights.some(item => item.keyName === 'login')) {
ctx.body = {
statusCode: '1',
errorMessage: '不具备登录权限'
};
return;
}
ctx.session.user = {
id: user.id,
roles,
rights
};
return githubUser;
};
Pay attention to the above code. If it is the first authorization, the user will be created. If it is the second authorization, the user has been created.
When the system is deployed or run, some data and tables need to be preset, and the codes are in app.js and app/service/startup.js
The logic is that after the project is started, use model to synchronize the table structure into the database, and then start creating some basic data:
After completing the above, the initial data has been completed and can operate normally
I bought the server centos on Tencent Cloud, the domain name I bought on Alibaba Cloud, installed node(12.18.2), nginx and mysql8.0, and started directly on centos. The front-end uses nginx for reverse proxy. Due to limited server resources, there are no automation tools Jenkins and Docker, which leads to some manual operations when updating.
